Announcement

Collapse
No announcement yet.

Home Automation - General Area, Issues, and Concerns ...

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Home Automation - General Area, Issues, and Concerns ...

    now that the structure of the house construction is completed and i have done most of the critical electrical and plumbing works of the house its time to get back to this major project i have started last year. this has been in the back burner for the past 6 months since i was busy with our house construction. as planned, i intend to showcase this DIY Home Automation System (HAS) in and out of our house to show that's its doable given enough passion and determination to see it through :-) he he he ...
    first thing to do is to review all of my notes to determine where i left off. then get all of my electronic stuff (temperature controlled soldering gun, temperature controlled hot air gun, multi tester, and all of the components that i have bought for this project) from the "baul" where it has been in storage for the past 6 months :-) he he he ...
    to manage the documentation (very critical to any major system development) of this project and share it with fellow DIY members, i'll start a new thread on each of the major areas of the project ... as much as possible, i'll try to reference this to my old post/replies last year (e.g. Home Automation - Open Project, Manually Soldering a QFN24). for now, below are the areas that i have already identified before based on where i left off 6 months ago ... cheers

    - general area, issues, and concerns
    - development and testing environment
    - security and encryption
    - hardware main host
    - hardware mid host
    - hardware end node (sensors)
    - wireless communication of nodes
    - wired communications of nodes
    - main host controller design
    - main host controller operation
    - main host controller communication
    - mid host controller design
    - mid host controller operation
    - mid host controller communication
    - sensor end-node design
    - sensor end-node operation
    - sensor end-node communication

    Boo!


  • #2
    i finally setup a temporary work area in the house for this HAS project. it took me awhile to find all the stuff that i need from the LBC boxes i shipped out from SG before ... 2-weeks ago, i scouted all of the electronic shops in marbel to find out where to get the electronic parts that i need - there are only 4 shops in marbel. to my dismay, most of the parts that i need are not being stock :-(. it looks like most of the parts available here are all through-hole electronic parts. one shop told me to go to gensan and try my luck if they have the parts there. given this constraint, i have no choice but to do the prototype boards using whatever parts i can source here in marbel or gensan. looking at the bright side, if i can make a prototype using whatever parts that are available here, then for sure, other PHM members (especially in metro manila) interested in doing it, can also make it :-) he he he ... cheers

    Boo!

    Comment


    • #3
      lucky to find this website :-) he he he ... if you have zero knowledge about electronics and you want to learn it so that you can create some projects, then i would highly recommend that you download this e-book and use it as your starting point in understanding the fundamentals of each of the electronic part that you need in your project. it's a huge e-book with 6 volumes, and each volume has a lot of detailed chapter explaining each area. even if you don't like electronics, just download all the 6 PDF volumes and for sure, i know that you can share the materials to a friend who wants to learn about electronics :-) ...

      it was in this e-book that i am now able to understand how "operational amplifiers" works (Volume 3 Chapter 3) which is the building block of any electronic automation (e.g. control circuits, measurement, etc) using an analog and/or digital circuit ... FYI, for a newbie in electronics, like me, you may have to read it (each subject area) about 2-3x times to really understand it :-) he he he ...

      Lessons In Electric Circuits ; 6 volumes (DC, AC, Semiconductors, Digital, Reference, Experiments) and each volume contains a detailed explanation of the subject area. part of the explanation is to relate the fundamental concept into the real world so that the reader (especially those who have zero knowledge about electronics, like me) can better understand the principles behind it ... cheers

      URL=> http://www.ibiblio.org/kuphaldt/elec...uits/index.htm

      Lessons In Electric Circuits
      A free series of textbooks on the subjects of electricity and electronics
      Copyright (C) 2000-2014, Tony R. Kuphaldt

      Boo!

      Comment


      • #4

        Basics: HAS Addressing of each Unit ... Part 1 ... What its all about and Why it's Important and Critical ...

        a home automation system (HAS) is basically a collection of sensors (Node) and microcontrollers (MCU). sensors are basically 2 types, those that monitor (e.g. getting a reading) the environment and those that initiate an action (e.g. turn on/off). for the sensors to work as a team, it needs to be manage. this is the role of a microcontroller. MCUs basically has 2 roles, (1) interact with the sensors it has been assigned (if any) and (2) interact with other MCUs in the system. the key to remember here is that each unit (sensor or MCU) has a specific role to play in the HAS team.

        the best way for an ordinary person to understand the concept is to relate it as an example in a typical work place environment, like an IT development team, which i assume everyone is familiar with as it's just like any ordinary work place :-).

        the programmers (staffs ; sensors), the basic unit and most important :-), doing the things its team leader (supervisor ; MCU that control sensors) requires - e.g. one development team (monitor sensors + MCU-1) may handle batch programs and another team may handle online programs (action sensors + MCU-2). you might also have another team of programmers that do online and batch programs (monitor sensors + action sensors + MCU-3). the supervisor tells the staff what needs to be done.
        for the different team leaders (supervisors ; MCU-1, MCU-2, MCU-3) to work harmoniously, again someone has to manage the teams. to expand our example, this will be the role of a systems analyst (assistant manager ; MCU-4). it's main responsibility is to prevent any issues (e.g. pag aaway ng mga supervisors) between each teams. the SA (MCU-4) will not be concerned with the programmers (staff ; sensors) in each team as he/she knows that the team leaders (supervisors ; MCU-1, MCU-2, MCU-3) are there to manage them. which is why, no programmer (sensors) is assigned to him/her. you can think of the SA as being the second level of management (mid-host) - coordinating everything between the teams (supervisors) and senior management (manager). the team leaders (supervisors ; MCU-1, MCU-2, MCU-3) are the first level of management.
        just like in any IT development project, you will have a number of SA with their own area of specialization. all of them will be reporting to a project manager (manager ; main-host) who's main responsibility is to inform the stakeholder/s (users of the system) of what is happening in the project.

        just like in a team of working people, you need a way to know each member of the team - their names :-). in any application system (e.g. HAS), each unit (sensors, MCUs) is identified by its own unique address/key (e.g. name) so that a person/unit can communicate with another person/unit in a team/system. in the real world, if you need an info from a person, you communicate directly to the person that you need this info - e.g. if my boss said, Boo!, i need the result of your analysis, i will respond by giving him the result of my analysis :-) he he he.
        in an HAS system, it's the same thing, if MCU-1 needs a status from Sensor-3, MCU-1 sends a message (get a reading) to Sensor-3 asking for the status and Sensor-3 will reply with the status.

        in any IT apps, without a unique address/key (name), 2 things cannot be done, (1) information specific to each unit cannot be stored in the system, (2) no communication can be done with the different units in the system. without this two, information generated by one unit cannot be retrieve and process by another unit to initiate an action based on the result of that information - basic principle of an HAS system, IF result is this, THEN DO this action.

        it is therefore critical (the most basic thing) that the system design incorporates a good addressing logic to identify all of the units (and where it's info resides) in the system taking into consideration both the hardware and software limitations that you will be using to run the system.

        doing this, a person using the system, will be able know the status of all the sensors in the system and the MCU controlling them and all the other MCU controllers keeping the system to run harmoniously in monitoring the activities inside and outside your home.

        next posting, i'll discuss the things (hardware and software limitations) i considered in the design of the addresses of the units in the HAS that i am developing right now. the good thing about doing this activity, besides sharing it to the group, is that, it already forms part of my documentation of the system. right now, all of my design notes are scattered everywhere in my notebook. its quite time consuming just to remember what was on my mind when i made that note of the design, so it's better to keep up with the documentation while i'm doing the development ... matanda na talaga ako :-) he he he ... cheers

        Boo!

        Comment


        • #5

          Basics: HAS Addressing of each Unit ... Part 2 ... Things to Consider in a Good Design ...

          having a clear understanding of a unit's unique address and it's importance to HAS in the previous post, it's time to look at the things to consider in designing a good unit addressing system logic. always keep in mind to make the design as simple as possible. a simple design means a lot if you have limited hardware and software resources to start with.

          this posting is all about knowing the info important in determining a good unique addressing logic or process ...

          1. before anything else, it's best to document the requirements of the HAS that you are going to build ... what you want it to do for your house :-) ...

          2. know the existing constraints in the hardware / software you will be using and then work the design with it and not against it. right now, for most DIYers familiar with programming, the best option is to develop an HAS using a prototyping platform like arduino (there are also other platforms in the market which you can look at). why? because a lot has already been done by others in the past, which will make it more easier to make your own HAS. from my research experience, 80% of the time, most of the ideas i have come up, has already been look at and done by someone in some sort of way.
          if you plan to use arduino (like me), then you should know the constraint that would limit your addressing. good if you can think of a work around to the constraint. if not, you need to live with it :-). some of the constraints related to storage and communication (critical to unit addressing) which i have noted so far while developing the system are:

          on communications ...
          - you only have 2 ways of communication - wired and wireless and both needs a unique address for each unit for it to work.
          for wired communication, the most available method already built-in into the MCU is to use SPI (serial peripheral interface) or I2C (inter integrated chip). for my case, the wireless communication (NRF24L01+ on 2.4GHz) i plan to use can only interface with an MCU using SPI. so for wired communication, i have no choice but to use I2C.
          - for communication, the minimum and most common address size to uniquely identify each unit is 1 byte. always use 7 bits instead of the max 8 bits for the address as most communication interfaces uses 1 bit for it's own processing. NRF24L01+ uses 3-5 bytes for it's addressing, so using a minimum of 1 byte should not be a problem. with 7 bits, you can uniquely identify 128 units in a system. for I2C communication, only 112 (values 8 to 119) unique addresses is available for use as the other values is use in its own internal process.
          - for my case, with a setup of a main-host and some mid-host as additional units in the system, of the 112 addresses, only 96 unique addresses will be left for end-nodes and sensors (e.g. 48 end-nodes with 1 sensor connected to each). 96 would be ok if we are only talking about end-nodes, but if we include the sensors, then it is really too small for a maximum number.
          - if 7-bits is not enough, then we could use a next higher level, like 12-bits address - by just using 11-bits, it could already handle 2047 (0 to 2047) unique addresses. this is more than enough for a typical home HAS.

          on storage ...
          - most MCU either has 512 or 1024 bytes of storage included in the MCU chip for your data. for my case, i prefer to design a system with the most limited resources so that migrating to a bigger hardware will be simple. with 512 bytes and each unit having a 1 byte storage, then maximum number of units you can have in a system will be 512. with 2 bytes of storage for each unit, that gives you 256 units. with 1024 bytes, you just double the number of units you can handle with 512 bytes.
          - as you can see, whatever amount of storage (e.g. smaller storage means more units in the system) used per unit in the system, the 12-bits addressing will be able to handle it with no problem. what needs to be done is to check how many units has been defined in the system at any time and then give you a warning if it's nearing the limit of your capacity. then you may want to consider adding an EEPROM to the unit to increase your storage :-) he he he ...

          3. keep the design simple and efficient. why? because the MCU are not as fast as your computer/notebook CPU :-). a simple and efficient design means faster processing.
          at worst scenario, the HAS should be getting sensor readings from the environment every minute - e.g. in an LNG plant i worked in the 80's, the readings was taken every 10 seconds and displayed in a GUI screen, an average of the readings is done every minute for upload to the host. now, this things can be done by an MCU provided that the program is simple and efficient.
          for action sensors that needs to initiate an action (e.g. open light) based on a change in the environment (e.g. at night when a person enters a room), the response time should be instantaneous.

          to further improve processing time, spread the task/activities to be done all across the MCUs at all levels of the HAS so as not to burden each MCU. for my case, this was my plan when i included mid-host units in the HAS that i'm developing.

          for my case, i'm just using a simple tree (e.g. from 1 trunk it branches out like a tree) topology design for the HAS network ... where the main-host is link to a communication line ... 6 mid-host (max) are in the line ... each mid-host controls 15 end-nodes (max) ... and each end-nodes controls 15 sensors max ... doing the math, for each line, you will have a maximum of 90 end-nodes (6*15=90) and 1350 sensors (90*15=1350) ... if you want more capacity, then just add a new communication line. the number of lines is a global system parameter that you initially set to 1 and just update it every time you add a new line ... the addressing to reach each unit in the HAS only has one logic that applies to all the lines.

          the good thing with this simple design is that it makes the system scalable with little or no change to the HAS application (core) software - just a change in one parameter. i learned of this technique in the 80's working in the LNG plant - using fortran language for the codes. the technique still works after 30 years :-) he he he ... cheers

          Boo!

          Comment


          • Boo-Semi-Retired
            Boo-Semi-Retired commented
            Editing a comment
            just an update on this project of mine ... the problem (handling a huge number of events/rules for a base configuration with a max of 1350 sensors) is still unresolved as of now. the problem is basically due to the limited memory/eeprom space in the MCU plus the fact that users must have a freehand (based on design) in setting up their events/rules so long as they stick to the system limit - 1 event can have a max of 8 input conditions (rules) and a max of 8 output actions (rules).

            i have explored and tested a number of possible software options to resolve the problem but the results are not really that satisfactory, even for my taste. so it seems that the only option left for me is to look at a major change in the design strategy - i was trying to avoid this as it will involve a lot of changes to the HAS system.

            the original design was to distribute the process throughout the HAS network to ensure quick response time - end-nodes to handle the sensor reading, conversion, and actual actions to events ; mid-host nodes to handle event/rules processing (this is where the space problem exist) ; main-host node handles the consolidation, batch / housekeeping process, and initiation of actions.

            after weeks of reviewing and analyzing the process and the existing programs, the best approach to take are:
            (1) also distribute the event/rules processing across all the nodes (MCU),
            - if both monitor and action sensors for an event/rule is controlled within a particular end-node, then handle it within the end-node process.
            - if the monitor and action sensors for a particular event/rule are controlled by different end-nodes which is controlled by a single mid-host node, then handle it within the mid-host node process.
            - if the monitor and action sensors for a particular event/rule are controlled by different end-nodes controlled by different mid-host nodes, then handle it within the main-host node process.
            (2) add a new validation routine in the processing of the parameter setup text file and compute that the amount of memory/eeprom space required for all the files/tables required by HAS does not exceed the limit of the individual node hardware in the HAS network.

            next step now is to document all the changes that needs to be done to each of the HAS programs/functions/libraries and the changes to the system global parameters. once completed and verified to be correct, then actual revisions to the system can be started ... end result, a lot of sleepless nights for the next month or so :-( ... cheers

            Boo!

        • #6

          Basics: HAS Sensor-Action Relationship ... Part 1 ... Things to Consider in a Good Design ...

          once the design and the logic of the unit addressing for communication and storage has been defined, the next most important aspect to consider in a good HAS design is the logic for the relationship between sensors and the actions to be taken in an HAS.

          i realize this before while testing a stand-alone end-node program. it suddenly hit me, it's not going to be a simple task (storage and logic) to setup the controls for the action to be taken based on a sensor reading, if your talking about hundreds of sensors and it's possible iteration of actions to be taken ...

          in simple terms, how do you work out the concept of "if-this-then-do-this" in the HAS for all the sensors (monitor and action) to work together. well, it's all about control ... come up with a simple and yet effective control design to ensure that the HAS will do the right thing at the right time. if your going all out, ika nga, todo ambisyon, then it's a big undertaking for a single person to do - the difference between a low level and high level HAS are based on the extent of control provided, the degree of feedback, the level of “intelligence” and the sophistication of the user interface.

          so to keep my feet on the ground, the design should be somewhere in the middle, more to the low level :-) he he he ... taking note of the following ...

          (1) you can only automate a process up to a certain extent. why? because computers are not human, it lacks something that humans have - perception, being aware of something base on ones senses and knowledge. to complicate matters, humans, being unique individuals, will at times, react differently, even in a non-logical way :-). a computer, on the other hand, only uses logic for control, '0' or '1'. no gray area in between, it's either 'this' or 'that', no in-betweens.
          so, at some point in the process of setting up the rules, it's more practical to simply ask the user what he/she wants to do :-), rather than come-up with a complicated logic to determine what action to be taken based on a certain condition.

          (2) no need to do the user interface (e.g. borloloy ika nga) for setting up the rules - this can be done at a much later time once the HAS is up and running. simply use a text file in a format that can be process by a simple program to setup the tables/files for the control of the HAS. the critical thing is the proper setup of the relationships of the tables/files with each other to implement the concept of "if-this-then-do-this" in the HAS control process.

          as what my mentors (project managers / system analyst) before told me when i was then, just a programmer, the secret to a good design is to break down the scenario into little things (process) and determine the 'missing link' ika nga, that relates all of it together - the more basic it is, the better the design will be.
          it takes time to analyze and see (understand) it, you just don't get it in one sitting. in my case, it took me a few weeks of stress free gardening (e.g. mag bunot ng mga damo) to get a workable idea. yung creative thinking ko comes out when i'm doing routine (no need to think) manual labor that i enjoy - helps me think of something else :-) ha ha ha

          ----------
          concept ... 'if-this-then-do-this' ...
          ----------
          can be express as ... monitor sensor => action sensor ... or it could be monitor sensor = action sensor
          ----------
          from that, a final relationship ... monitor sensor <= event => action sensor
          ----------

          the key to the basic relationship of a sensor to an action is the 'event'. the 'event' is not the result of the sensor being use by the HAS or the action taken by the HAS. it is the reverse, the 'event' dictates what sensors to use and what action to take - this is a more logical relationship, and it makes more sense of describing an actual scenario, rather than the other way around. this, i think, is a much better design for a rule-base method that is commonly use in an HAS.

          all of the programs/system i have read about HAS in the internet, points to a basic design that uses the sensors as the primary (basic) mode of controlling the system and what action to take. it works, because it should work, as can be seen in their relationship - especially true for stand-alone module.
          but, such a design, makes it difficult and complicated to handle an 'event' using multiple sensors with multiple possible actions to be taken depending on the status of the sensor.

          using the 'event' as the key to relate the tables/files use to control the system will result in a much more simple table/file design and most importantly, a simple logic to follow :-) ... consider this ...

          on one (left) side of the equation are the monitor-sensor/s sending info to the host (main-host or mid-host) of their readings of the environment ... the info of each sensor is process and the status of each sensor is set to ON/OFF logic depending on the reading.

          in the middle of the equation is the event defined by the user thru some simple condition ... the host (main-host or mid-host) check the status (ON/OFF) of all the sensor/s related to a particular event ... if all the sensor/s status for an 'event' are the same (ON or OFF), then the event is considered to be happening right now. the OFF is the reverse condition of an event, in case you want to trigger something when an event is not happening.

          on the other (right) side of the equation are the action/s to be taken if the event is happening ... the host (main-host or mid-host) then instructs the end-node controlling an action-sensor to initiate the action.

          notice that the 3 items above as a group still follows the simple concept of 'if-this-then-do-this' :-) ... and it will be easier for the user to setup his/her automation rules.

          just to summarize how this posting (sensor-action relationship) relates to the previous posting (unit addressing) ... unit addressing let's you identify, specifically each of the units (main-host, mid-host, end-node, sensor) currently in use in the HAS ... this sensor-action relationship let's you define the role/s of of the units, like sensor/s in the 'event' that is happening at any point in time

          next posting, i'll show some examples of how to break down 'event' into simple things to understand how to group them to further fine tune the relationship between sensor/s and action/s ... cheers

          Boo!

          Comment


          • #7

            Basics: HAS Sensor-Action Relationship ... Part 2 ... Examples of Events ...

            in a home automation system (HAS), events are simply scenarios/settings that are happening inside and outside your home environment (within your property) that the system can detect to serve as a means of control on how the automation process is to be done ...

            a classic example of a simple but critical event is day-event or night-event. for obvious reason, a night-event is important because it is one of the 'rules' (there may be additional rules) you set for turning ON a light inside or outside the house. a light sensor (LDR - monitor ; place outside the house) is use to detect this event. and a relay (switch - action) sensor is use to switch on the lights (e.g. outdoor light), an action (base on rule) taken for an event that is happening as detected by the HAS.

            to further expand our example ...

            consider another set of events - bright-event or dark-event. a dark-event is a scenario (not well lighted) inside the house when there is less sunlight outside. it's basically the same as the above event (day/night) with it's setup - light sensor and a relay sensor. the only difference is on the placement of the light sensor (e.g. inside a room) and which light is controlled by the relay-sensor (e.g. room light).

            of course, the threshold setting (e.g. the value of the data at which point the sensor status is change to ON/OFF) in the rule of a light sensor (LDR) for day/night and bright/dark will be different - example, suppose the setting for night is 100K resistance, then for dark it could be like 80K resistance, resistance in the light sensor is higher as it gets darker.

            some notes on the above examples:
            - two opposite events can be monitored by a single sensor, each event has the reverse status (ON/OFF) of the other one.
            - the placement of the sensor is determined by the event - for day/night, light sensor is outside the house, for bright/dark, light sensor is inside the house.
            - what is not mentioned, but can be deduce from the relationship of the 4 events, is that its technically possible to only use 1 light sensor for the 4 events. if resistance <80K, then it's day and bright. if resistance is >80K and <100K then it's day and dark, and if resistance is >100K, then its night and dark :-).
            - and most important ... how it fits our model of the event dictating the monitor-sensor and action-sensor to use ... monitor sensor <= event => action sensor

            just like how important an address is to identifying a unit, events when properly setup in HAS means better control of the situation. so how do we group them to be more effective ...

            from the example above, one way is to group the events in terms of it's impact to your home - global or local. in addition, i will include another group, i'll call it, system event that i will explain later.

            global events in HAS means those events that impacts the whole property (inside and/or outside the house) in some way ...
            (1) if its current status (ON/OFF) also affect other events setup by the user and monitored by HAS - it means, it's also use as part of the rule for other events.
            example of this is the day-event or night-event above. it can be use in another event (e.g. room-1-in-use event) in which the rule can be like ... if night and room-1-motion-ON then room-1-light-ON ... when its night and someone is in room-1, turn ON the light.
            (2) if the event is a major event from the perspective of the user. as per the model, it is the user that define the events, which then determines the sensors (monitor and action) to be use. from my old notes, i listed some of the events (below) that i think is critical to a typical HAS setup.
            - Fire event
            - Gas (LPG) Leak event
            - Brownout event
            - Lock-Down event (no one is in the house)
            - No-Activity event (all asleep in the house)
            - Night event
            - Day event
            - Intruder Alert event

            local events pertains to events that happens only in certain areas inside or outside the house (within the property). in the example above, bright-event or dark-event are a good example as it only impacts a portion of the house (not well lighted) where it easily get's dark when there is less sunlight outside. most sensors will be in this group.

            another way of grouping events is by the type of sensors (e.g. LDR, PIR, reed, etc) used or by the number of sensors use in an event. local events that use motion sensor to detect movement in each area/room in the house can be group into 1 to represent an event for a bigger area. example, all the motion-sensors inside a house can be group into 1 event to denote activity or no-activity in the house.
            from my old notes, i did make a note of the technical possibility of grouping events using multiple sensors of different type (e.g. PIR, relay, etc) but i also noted that it needs further analysis and i can't remember what was my issue then :-) he he he ... once i remembered the reason, i'll update this posting.

            i earlier mentioned about system events. it's main purpose is to execute system related task like housekeeping activities (e.g. date/time synch up of all units, system reset, global update of parameters, error reporting, etc), backup activities, batch process, data mining activities for learning events based on logs and MIS generation. since this event tables/files are routinely monitored by the main-host to trigger an action when an event happens, it is also a good place to include system events in this process. once a system event happens, instead of initiating an action, it instructs another MCU to execute a task. so, whatever key is use for the events, a certain range must be reserved for system events use.

            given all of the above examples, the next task is to simply design/develop how the different tables/files related to event processing will interact with each other using an event-id as the main driver for the process ... cheers

            Boo!

            Comment


            • Boo-Semi-Retired
              Boo-Semi-Retired commented
              Editing a comment
              Update:

              ... from my old notes, i did make a note of the technical possibility of grouping events using multiple sensors of different type (e.g. PIR, relay, etc) but i also noted that it needs further analysis and i can't remember what was my issue then :-) he he he ... once i remembered the reason, i'll update this posting. ...

              i just remembered (while drinking coffee and reading the news) what was my issue before :-) ... it makes the program logic more complicated (e.g. managing I/O, managing the rules as to when action is to be taken, etc). this results in having a bigger program size in the end. one of the requirement was to limit the program size of the end-node to 8KB.

              in order to address it, my notes to the issue was, to distribute the other non-critical end-node process to other mcu to keep the program size small. the end-node should only do 2 simple task:
              (1) get a reading from all the input sensors connected to it. convert it to a meaningful information. send the information to the host.
              (2) if the end-node has action-sensors connected to it, the other task is to wait for, and receive the host request and act on the request. ... cheers

              Boo!

          • #8
            Click image for larger version

Name:	Event-Rules-Process.jpg
Views:	1
Size:	73.8 KB
ID:	84129


            Event (Rules) Processing ... Part 1: Understanding an Event

            found this photo from my old notes, it's the initial design i made for handling Event (Rules) processing in HAS.

            what is event (rules) processing? in simple terms, it's the heart of the HAS system, the most critical and important part of an integrated HAS. it relates an event that is monitored through a sensor, to an action to be taken by the system - it's the control logic implied in the "if-this-then-do-this" concept.
            it's also the most difficult part to do because it uses logic knowledge compared to factual and/or process knowledge for the other modules (e.g. reading a sensor value) of a HAS. it's also one area that is hardly shared/explained by others (who made a HAS) in the internet for one reason or another - parang taboo or trade secret of the people in the know.
            there are a lot of ways/methods of doing it from the most simple to the more complicated ones you see for hardware/software packages being sold in the market. each method has it's own pros and cons that goes with it.

            the simple way is to hard code the events logic into your program, normally seen in a stand alone HAS units. these are the things mostly done by enthusiast who likes to do things fast - aka, quick and dirty setup that works, if the event condition does not change. one disadvantage of this is obvious, and change/s to the event condition will require changing the program of the unit.

            on the other side of the spectrum, are the commercial version being sold in the market, it has all the bells and whistles, like a GUI interface for the user to change (update, add, and delete) the event condition/s via a smartphone or computer connected to the HAS and in a matter of a few seconds, the system will already be executing your changes. one of the disadvantage of this is the cost (so expensive). in addition, it (only) always work with the specific hardware that they also sell to you :-). it means once you invest in it, your tied to it for eternity.

            as i have always mentioned before, as a DIYer, we go with a method somewhere in the middle :-) he he he - simple and yet flexible enough to accommodate the things that we want to automate inside/outside our home.

            the top end HAS application software/hardware you see in the market was most likely done using a rules engine to design the application software. rules engine software packages are quite expensive (it makes your program development really simple) and you need to find the perfect match to your requirements - rules engine are not made equal, don't go for the low ends ones at most likely palpak yun :-). a good example of an application software that uses rules engine for it's design is the CI (credit investigation) module use by financial institution to provide a credit rating value of their clients/customers based on a number of demographic information/data that a customer provides in the application + all the other info coming from other credit bureaus.

            implicitly implied in the above methods (simple to complex) is the number of sensors a system can handle and how fast is the process, to react to an event with an action - more expensive means more sensors and faster response time. example, the design i'm working on can handle a max of 1350 sensor units for a basic setup. if your getting a reading from all of it (sensors) every minute (worst possible scenario ; ideal is around 10-15 seconds), your looking at getting a reading from 22-23 sensors every second (1350/60=22.5) on average. if your event processing is slow, the delay (milliseconds or seconds) in initiating an action to a change in the event will be high. the "high" delay will always have a certain limit that is acceptable to a user - e.g. when you walk into an area (room or outside) at night, everyone would want the lights to be ON asap, no delay. a delay of about, say, 3-5 seconds should still be OK to most users. but a delay of like 15-30 seconds is no longer acceptable to most users :-) he he he. so, you see how important that an event process is fast and efficient especially when you have a lot of sensors connected to the HAS.

            so, what makes a good design for an event (rules) processing module ...
            from my experience, i can summarize it in this statement ... (1) having a good basic understanding of an "event", you can (2) design (simple) the table/file relationship/s to store the information for those "event" and come up with a (3) logic that will give you access to those information when your program needs it for it's processing while (4) taking into consideration the limitation of the hardware setup in which your application program will be running ...

            (1) having a good basic understanding of an "event" ...

            in my other posting, i mentioned the basic model of HAS in terms of sensors and event ... monitor-sensors <= event => action-sensors ... with this in mind, it follows that an "event" can only be in 2 state - ON (1) or OFF (0). the monitor-sensor tells the system when an event is ON or OFF based on it's reading. at the same time, the action-sensor is the one that "acts" for the system by doing an action if the event is ON or OFF.

            so the control of the whole process lies on the status of an "event". in math, for one to know the status of x (something), you need to compare it with something (another thing), a value that has some relationship (condition) with the status that you want to know ... in effect, you just made a mathematical condition, like x = 10 or x >= 20, etc ... so an "event" is simply a condition (comparing a value with another value) that you describe as an "event", whose result can only be either TRUE (ON) or FALSE (OFF).

            a simple example, an event that is Night. night is normally measured by the amount of light in an area. Lux is a unit of measure for the amount of light in an area. normally, a 10 Lux unit means that it's already night time (quite dark). so if i have a sensor reading (raw data) from an LDR that i convert to a Lux unit, i can then create a simple condition for a night event like:
            Lux reading <= 10 ... if true will mean that a Night event is happening

            as shown above, an event can be translated into a simple single condition. it also means that you only needed 3 things (information) for each condition, (1) is the sensor reading converted to a meaningful information like a unit of measurement, (2) the conditional operator which you use to show the relationship, which is only 4 types (e.g. =, !=, <=, >=), and (3) a parameter value for comparing it with the information from the sensor reading.
            in reality, an actual event will normally involve multiple (two or more) conditions. this can be easily express in the logic by entering one or more condition for an event and each condition must be TRUE for an event to be TRUE.

            the only thing that limits the system to detect any event with multiple conditions is the issue of how many conditions can the system handle (monitor) in 1 event?
            well, that depends on the design of the system. as i have mentioned before, it's better to start small and see how the system works - useless to design a system that can handle an event with a 64 conditions max if the hardware cannot handle it.
            1 byte is the smallest unit of data that a computer normally handles. a byte has 8 bits (bits 7-0) and each bit can represent 2 state, represented by a 0 or a 1. so a bit is a good way to represent a condition if its true (1) or false (0). this means that in 1 byte of data for an event, we can handle 8 conditions independently. i think 8 is a good number to start with as an initial limit to the number of conditions one can use in one event ... it gives the user some space (but not too much) in defining his/her events in HAS :-) ... and considering that you might only have about 512 bytes of storage to work with, its better to start with a small capacity ...

            on my next posting, i'll explain the relationships of the tables/files as shown in the photo above. for IT people, they should already have an idea of what i'm taking about :-) ... cheers

            Boo!
            Last edited by Boo-Semi-Retired; 12-03-2015, 07:06 PM. Reason: change the photo size to a lower level

            Comment


            • #9

              Event (Rules) Processing ... Part 2: Design for the table/file relationship/s

              (2) design (simple) the table/file relationship/s to store the information for those "event"

              as mentioned in part 1, the 3 things you need for an event condition are the reading info, conditional operator, and the parameter info for comparison. to tie them all together, we need another file, which we can call the event file.
              in a computer you have gigabytes (GB) of space for memory (e.g. 4GB) and storage (e.g. 500GB) to work with. for embedded systems like a HAS, you have very limited space for both. program memory for an MCU normally comes in 8KB, 16KB, 32KB (kilobytes). for storage, its 512, 1024, 2048 bytes. for my HAS project, the final MCU that will be in production will only have 8KB of program memory and 512 bytes of storage - really not that much to play around with :-) ha ha ha.
              which is why it's very important to maximize the use of this vital resources by keeping the table/file design simple to reduce storage to < 512 bytes and the program logic to only what's necessary to keep it small and fit the 8KB size.

              another thing to note with embedded systems, given the limited storage size is, you can't use (even if we wanted to) any index files to directly access any of the information. the only recourse is to use direct addressing through the table handling facility available in the program language (e.g. C/C++ for Arduino) you will use to code your program. as such, pointers (location address of the information) has to be use to directly access any information you will use in your program.
              location addressing makes the program logic much harder to code, but not impossible :-). the important thing to take note is to always ensure that the integrity of the location address pointers are always intact, otherwise unpredictable results (program bugs) will happen.

              the event table is a logical starting point for an event processing. it's primary role is to point where your information are located - pointers to the address of the info in the Sensor Input (monitor sensor) and Sensor Output (action sensor) tables ...
              ... as i have always coached people starting in IT, think of your self as the "logic" that is walking through a maze with lots of crossroad/s and turn/s along the way, where at each step you are performing a task based on the information you get as you walk through the maze. once this thinking becomes natural to you, for sure, you will excel in any of the IT fields you choose :-) he he he ...
              since we are using byte addressing as a starting point to know the location of each info, the max events that can be handled by the HAS will be 255. i think this should be more than enough for our purpose right now. this could easily be increase later if needed.

              the 2 tables at the center are the tables where you have most of the info you need for event processing. one is for input which will contain all of the conditions that needs to be tested to determine the current status of an event - ON or OFF. another one is for output which will contain all of the conditions for an action to be taken depending on the status of an event. the 2 tables are basically the same. technically, you could combine the 2 tables into 1.
              but why design it this way?
              if you really analyze the basic relationship of an event to a monitor-sensor (input) and an event to an action-sensor (output), you will notice that although they are similar in most aspect, they do have some differences.

              the conditions you check for an input sensor is different from the conditions your checking for an output sensor. for input, you check if all the conditions are TRUE to set an event as TRUE. for output, base on the status of the event and the current status of the output sensor, you determine whether to do nothing (no change in status) or to turn-OFF or turn-ON an action sensor due to a change in the event status.
              an event needs at least one monitor-sensor for you to monitor it. it does not make sense (illogical) to define an event in which you have no way of getting a reading for it :-). people may think time event is an exception, but it's not. i'll discuss this (date/time event) further in some other postings as it's not part of this topic.
              an event does not need to have an action-sensor associated with it - you must monitor an event but it does not mean you always need to do an action for it. example, getting a temperature reading inside and outside the house - you need to know it but you don't need to do an action for it. another event with this condition can do the action. i'll discuss this (pseudo-action) further in some other postings as it's not part of this topic.
              so, you see, based on the relationships above, it makes more sense to separate the input from the output rather than combine them into one table - the design reflects a better representation of what's happening in the outside world :-).

              in the 2 tables, you notice a summary record and condition/s records following the summary record. this is one method of handling the actual scenario of not knowing how many conditions the user will use for the event that he/she wants to monitor and possibly take an action for it. we know the design can handle a max of 8 conditions. but what we don't know is how many the users will use - from 0 to 8. thus, the summary record will contain an info (counter) of how many conditions an event is using for it's input and output. you can store other info in this record depending on the process you want to do.

              the condition record/s that follows the summary record should have all the pointers needed for the information you need to determine (On/Off) the status of the condition. this is why you see it's pointing to the parameter table and the sensor info table.

              it's important to highlight that the sensor reading should be an information and not a raw data reading - there is a big difference between the two. raw data is the just the sensor's hardware measurement of the event. it is important that such a reading be converted to a meaningful information for which the hardware sensor was made for. there are lots of information in the internet that gives you the formulas of how to do this - e.g. converting millivolts to Lux, which is a measure of light.
              why is this conversion needed?
              because it makes the internal process of the HAS application system independent of the sensor hardware - you can use any sensor hardware to get a (raw data) reading from the environment and by just handling the interface/conversion of it to a meaningful information, you can forget about how the hardware is actually getting the raw data. it's similar to the plug and play term that you normally see in a lot of the computer accessory products you buy.

              lastly, you need to understand that the above is not the only solution out there to handle an event process and it also may not be the best solution. somewhere out there, another person (need not be an IT person ; a lot of innovative design comes from people in other fields who thinks outside the box if they understand the problem on hand ; paradigm shift, looking at an issue from a different prespective) will have another way of doing things.

              i got a chance to learn of this design in the 80's when i was responsible for maintaining the application system of an LNG plant automation process running on an HP-1000 midrange machine with PLCs (programmable logic controller) as the sensors. it took me a few months of reading and understanding (you always ask yourself, why did they do it this way?) hundreds of fortran source codes to finally get the idea of the basic design. sayang nga, all my notes then was stored in an Iomega zip disk that i have thrown away a long time ago. i just realized now how important those notes are when i started this HAS project :-( ...

              a plant automation system is basically just a home automation system on steroids :-) ha ha ha ... next posting, i'll explain the 3rd item which is about the process logic for an event processing ... cheers

              Boo!

              Comment


              • Boo-Semi-Retired
                Boo-Semi-Retired commented
                Editing a comment
                update on the event processing to include pseudo action event ...

                i mentioned in previous posting about pseudo action event. but what are this type of events? the term &quot;pseudo&quot; means &quot;not real&quot;, so basically it's an event with no real action sensor associated with it - e.g. turn-on light, start a motor, power to an AC outlet, etc. people may ask, why then monitor an event if you don't intend to do any action for it once it's happening (ON)? well, there is an action, but it's not a physical action of activating/deactivating an output sensor, but more of simply keeping the information of the event for informing the user of what's happening in the home automation process.

                a good way to picture how this pseudo event works is through an example. suppose a user wanted to know what is the temperature inside and outside house. to do this, you install 2 temperature sensors one inside and one outside - you can even install multiple sensors and get the average of it :-). once the sensors are in place, you setup an event like getting a reading of the temperature every 1 minute. this type of event can be seen as a pseudo action event because it simply gets a reading of the input and does not perform any physical action whenever the event happens (every minute) - the resulting information is the action of the event which is why it's called pseudo :-).
                as to the information (e.g. temperature reading), it's usage is only limited by your imagination and how you want to use it for your home automation. the most simple use is to display the info to the user either in your HAS screen (laptop connected to your main host) or send the info to OpenHAB.org so that you can display it in your smartphone connected to the internet.
                a more complex setup could be to use the info reading to control other events in your HAS - remember the input condition formula ... (reading) conditional operator (value) ... the reading or value can be the info that is the output of this pseudo action event depending on how you setup the input condition of the other event.

                the concept may be simple but it is an integral part of the basic foundation of a good automation process whether at home or any place for that matter - using information to control the process ... cheers

                Boo!

            • #10

              Event (Rules) Processing ... Part 3: Processing Logic

              (3) logic that will give you access to those information when your program needs it for it's processing

              if you have done item (1) exhaustively, giving you a good understanding of an event and it's relationships to the input and output sensors. and based on this, designed a good tables/files relationship for item (2), then the logic (process) for the event processing will be quite straight forward ... its just putting into words the step-by-step actions of what you have been "thinking" in the previous 2 items.

              below is a pseudo-code of the process logic. a Pseudo-Code is simply a representation of the logic that you want to put into code using the programming language of your choice. a pseudo-code is a much better way of explaining the logic rather than actually showing a source code of a program.
              why?
              because of it's emphasis to the logic of the program which is more important rather than the task of coding it. pseudo code can be easily understood by people who are not familiar with programming - e.g. it's a good way of explaining a system to management in a layman's term that they can understand to get their support :-).
              another way of showing the process logic is thru a Flow Chart, it's more tedious and a bit hard to comprehend if the process is huge. but if you can condense the flow chart into one power point slide, it's a more effective way of presenting it to people (management) - it has been proven scientifically that a person can easily digest (understand) a diagram rather than words :-) ...

              personally for me, pseudo-codes are the best. it gives the reader/s, especially an IT person starting to learn programming, the opportunity to learn on his/her own, how he wants to do it after understanding the concept and the logic - it's an effective way of coaching. showing a source code as an example, is like enforcing to others that this is how it should be done. in effect, you are simply hindering the creative thinking of a person - rather than think of a better solution on his own, most of the time, the person will just copy your program and use it ...
              ... eh kung mas matalino yung tao sayo? :-) eh di na sayang yung talino niya at di niya na gamit, kasi tinamad ng gamitin at mas madali ang kumopya na lang :-) ha ha ha
              seriously speaking, this was how i was coached by my mentors when i was just starting in my profession and it's the most effective way of bringing out the best in a person. and doing it continuously on my own, for others to learn and maybe become even better than me, is my way of saying thank you to my mentors :-)

              going back to the topic on hand :-), for a person familiar with programming, below is a pseudo-code of an event processing for a HAS ... i recommend to always use a top-down approach to the logic to make it structured and easy to follow and maintain by the next person who will enhance it :-) ...

              ========== ********** ==========
              ========== ********** ==========
              start - process all event records ...
              (NOTE process for sensor input ...
              - get and save input pointer ...
              - get and save input counter from summary record ...
              - get and save the input byte status of all the conditions from summary record ...
              - initialize (all bits = 0) input status byte - 8 bits, each bit for each input condition for this event ...
              - set unused bits (bits > counter) to 1 ...
              - set index = address of next record in the sensor input file ... input pointer + 1
              - start - process all input conditions (index <= counter) ...
              - get all info needed for input condition testing ...
              start - do case ... test condition if TRUE depending on the conditional operator
              case EQ (=)
              - if reading info = parm info ... set corresponding bit to 1
              case LTE (<=)
              - if reading info <= parm info ... set corresponding bit to 1
              case GTE (>=)
              - if reading info >= parm info ... set corresponding bit to 1
              case NE (!=)
              - if reading info != parm info ... set corresponding bit to 1
              end - do case ... for each conditional operator
              - end - process all input conditions ...
              - update sensor input status byte ...
              - save byte info in a work area for use in next process to check output action-sensor
              (NOTE if all conditions are true (ON) for an event, the updated byte will have a value of decimal=255 ; hexadecimal=xFF ; binary=b11111111
              otherwise, value could be anything (OFF)

              (NOTE process for sensor output ...
              - get and save output pointer
              - get and save output counter from summary record ...
              - get and save byte status for all output action-sensor current status ...
              - set index = address of next record in the sensor output file ... output pointer + 1
              - start - process all output conditions (index <= counter) ...
              - get corresponding bit to determine current status (ON or OFF) of output action-sensor
              - get output action condition to check what to do next by comparing with other info on hand
              - if bit = ON
              start - do case ... check if ok to turn-OFF output action-sensor
              case ... if input status ON and output action is ON
              don't do anything, exit
              case ... if input status ON and output action is OFF and no-delay
              - call routine to send command to end node to turn-OFF action sensor
              - set corresponding bit to 0
              case ... if input status ON and output action is OFF and with delay
              - call routine to send command to end node to turn-OFF action sensor with delay parameter
              - set corresponding bit to 0
              end - do case ... check if ok to turn-OFF output action-sensor
              else (bit = OFF)
              start - do case ... check if ok to turn-ON output action-sensor
              case ... if input status ON and output action is OFF
              don't do anything, exit
              case ... if input status ON and output action is ON
              - call routine to send command to end node to turn-ON action sensor
              - set corresponding bit to 1
              end - do case ... check if ok to turn-ON output action-sensor
              - end if statement
              - end - process all output conditions ...
              - update sensor output status byte ...

              - end - process all event records ...
              ========== ********** ==========
              ========== ********** ==========

              as you see from the above, having a good basic understanding of what's happening will lead to a simple logic of the program. the above can still be optimized once you start coding your program to make it compact and efficient.

              as to item (4) taking into account the hardware limitation into the design, i think there is no need to further explain this as i have already mentioned this aspect in explaining the 3 items in my previous posting. just always remember this note ... design the system with the limitation on hand, let the design work with the limitation rather than against it ... cheers

              Boo!

              Comment


              • Boo-Semi-Retired
                Boo-Semi-Retired commented
                Editing a comment
                just an update on the event processing logic ...

                added time as another form of input condition for validating an event to be happening or not (On or Off). this was part of the initial specification for the event processing logic but the strategy for how to handle it in a simple and logical way in the application design was a bit elusive in the initial design phase. it only became clear to me this last 4 weeks when i was doing the SIT (system integration testing) and stress testing of the communication module - handle a max of 1350 sensor messages from 91 host at any point in time.

                in layman's term, time in event processing, can be treated as an input whose condition can be tested just like any other sensor input condition as i have explained in my previous posting.
                if we treat time as an input sensor, it means that we need to have a reading or an interrupt for it. we can look at the reading as elapse time, the difference between the start time and the current time. start time can be taken at the point when an event changes state (e.g. from ON to OFF or from OFF to ON) and current time will be the time at which the system checks the input condition. and the good thing about a time input condition is that you don't need a physical sensor for it. the chip (MCU) already has a built-in module to increment it precisely in terms of milliseconds (1/1000 second) from the time the MCU is powered up.
                the change in time (elapse time) can be express in a conditional statement ... (reading) conditional operator (value) ... where reading is elapse time (current time - start time) in units of time (e.g. sec, min) and value is some fix duration in units of time (e.g. sec, mins, etc).

                some examples of home automation event/s with a time input condition:
                1. at night, turn-on outdoor light when someone is in the area. when the person leaves the area, turn-off outdoor light after 15 mins.
                2. at night, when someone enters a room, turn-on the night light. when the person is still in the room after 10 sec, turn-off night light and turn-on main light.
                3. after a brownout, delay turn-on power to specific AC outlet by 3 mins (e.g. to protect sensitive appliances from power surge).

                as you can see from the examples above, having time as an input condition for an event can make the home automation more reflective of how you want it to be done.

                i'll post a separate item on the results of my stress testing ... cheers ...

                Boo!

              • Boo-Semi-Retired
                Boo-Semi-Retired commented
                Editing a comment
                another update on the event processing logic ...

                in the initial design of the event processing logic, an event can only be happening (ON) if all of it's input condition is true (all ON). it works, but the problem with this design strategy is that it does not provide enough flexibility to the user to freely define his/her conditions based on the &quot;way he/she thinks&quot;. in my testing (SIT) i notice that i need to constantly change the rules of the event/s i'm creating based on the processing logic of the program. worst, if i still could not get the result that i was expecting, i have to change the setup of the HW sensor at the end node host - something you don't want to do in production (live usage of the HAS) as the would entail a difficult activity to the person using the apps. baka mag mura lang yung tao at sabihin palpak yung systema :-) ha ha ha. bottomline, the design and the logic needs improvement - from my experience, people in general do not accept &quot;change&quot; openly, they would prefer to stick to what their use too, even the way they think in a given situation :-) he he he

                consider this simple scenario ... you have 2 sensors that can do the following:
                - a motion sensor, someone in the room (ON) and no one in the room (OFF)
                - a light relay sensor, turn-on light (ON) and turn-off light (OFF)
                you can create an event that looks like this ...
                - when someone in room (ON), turn on light (ON)
                - when no one is in the room (OFF), turn off light (OFF)
                both events will result in the same outcome, but the 2nd event cannot be handled by the existing logic :-). it's like describing a glass filled with water halfway, some people will say it's half full, others will say, it's half empty. so the system should handle for both scenario.

                one (there are a lot of ways of handling this logic) solution to this dilemma is to add another info in the input condition to know what (the user) is testing in the condition - is it the OFF state or the ON state of the sensor. the info can then be compared with the result of the condition (ON or OFF). if both are the same, then that's the time you set the event to be happening or not.
                the same thing can be done for the output action of the event. with the info, you will then be able to know what action the user wants to do when the event is happening.

                the result of this change gives the user the flexibility to setup the input condition and the output action based on his/her mode of thinking :-) he he he ... cheers

                Boo!

              • Boo-Semi-Retired
                Boo-Semi-Retired commented
                Editing a comment
                an update to the use of time as an additional condition for processing an event/sensor and it's impact to the processing logic ...

                without the element of time, an sensor state (On/Off) will change instantly base on a given criteria, either from ON to OFF or from OFF to ON. if we include time in the equation as another criteria, then the change in state will not be instant but will be done after an 'elapse time' has occurred. after analyzing how time works in HAS this last few days, it becomes clear that 'elapse time' criteria can be looked at in 2 ways:
                a. as a duration - the time required for an sensor to be in a particular state (On or Off). basically, it's the 'how long?' in time units should an On/Off state last.
                b. as a delay - the time required 'before' or 'after' a state (On/Off) is change.
                in IT, the above (adding time to a change in state) is similar to the principle of FSM (if i remember it right from my training course before, it means finite state of a machine). in layman's term, FSM is just adding the time factor for an input event. a classic example of this principle is how the computer's mouse works for the computer to know what you want it to do - e.g. single click of mouse button, double click of mouse button, continuous press of mouse button, etc. the program simply determines how many times you press a button or continuously press it over a period of time (moving from one state to another state) and executes the corresponding action, it's that simple.

                the important thing in the FSM principle, is knowing what is the initial state and to what state it is moving or going. if we apply this to a HAS sensor, it's initial state is either ON or OFF and it's moving to 'a particular state' will be dependent on the initial state - if it's ON, then it's moving to OFF, it it's OFF then it's moving to ON. how fast it takes to move from one state to another is the time factor (e.g. duration or delay) criteria included for that sensor.

                i listed below the following scenario for an output sensor in which the time factor is more use. for an input sensor, it should be the same but the use may be different. same thing can also be applied for an event which also have 2 states (on and off), in case you want to delay or control it's duration.

                1. unit is OFF
                - unit is OFF => ON ... unit is in an OFF state moving to an ON state with a time factor
                2. unit is OFF => ON ... unit in OFF duration is active and once it expire, the unit will be ON
                3. unit is OFF => ON ... a start delay (to set unit ON) is active and once it expire, the unit will be ON
                4. unit is OFF => ON ... an end delay (to set unit ON) is active and once it expire, and unit will be ON
                - unit is ON => OFF ... unit is in an ON state moving to an OFF state with a time factor
                5. unit is ON => OFF ... unit in ON duration is active and once it expire, the unit will be OFF
                6. unit is ON => OFF ... a start delay (to set unit OFF) is active and once it expire, and unit will be OFF
                7. unit is ON => OFF ... an end delay (to set unit OFF) is active and once it expire, the unit is OFF
                8. unit is ON

                Notes:
                a. without the time element, you only have 1 and 8.
                b. a delay can be either before (start) or after (end) a change in state has been detected. it really depends on how you setup the rules for the events.

                as an example, if you want outdoor light to be turned off 15 minutes after a person laves the area, the output condition to control the relay sensor for the lights can use scenario (6).
                another example, when a person enters a room, you want LED light to turn-on asap and then turn-off after 15 seconds and then turn-on relay sensor for the main lights if the person is still in the room. for the LED, the condition will handle scenario (5) and the condition for the relay sensor will handle scenario (3).

                in both examples, in an 'online' mode operation, the mid node host doing event (rules) processing sends an action command message to the end-node host doing sensor processing to which the sensors are connected. the end node host executes the action and provide the regular update to the mid node host as to the current status of the action ... cheers

                Boo!
                Last edited by Boo-Semi-Retired; 01-11-2016, 06:12 AM. Reason: remove the special characters use by PHM from the text ...

            • #11

              Event (Rules) Processing ... Part 1-3: Processing Logic - Major Re-Design of Logic

              an update on the testing of the event (rules) processing logic ... instead of just putting this as a comment to the previous posting, i posted it as a new item since this involves a major change to the original design of the event 'rules' processing ... new updates to the logic will be posted as comments to this item.

              with the 3 enhancement (below) that was incorporated into the rules processing, one off-shoot of the changes was that the program logic became complicated and as a result of this, processing takes too long which affect's the system's response time to a change in the event and the action to be taken. as i mentioned in my previous updates, a 1-3 second delay may be acceptable to most users, but anything greater than that will be outright noticeable to the user that the system is slow in reacting to a change in the event.
              - inclusion of the time element/factor in the input condition and/or output action of an event.
              - flexibility, what the user is monitoring
              - pseudo action

              after doing an in-depth analysis of where the problem is, the problem lies in the original basic design of the processing logic of the rules processing and the way the tables/fields was designed to support such a logic. incorporating the handling of the time element into the program, made the logic complicated because the original design was meant to only take a snapshot (current status) of the events at a point in time and then take the necessary action.

              as i said before, people normally avoid change :-). IT people (e.g. programmers, systems analyst, project mangers, etc.) are no exception - they will, as much as possible, avoid changing the core programs or the basic design because doing that means a lot of work. they would rather come up with complicated work around solutions to achieve the same result and let the next guy worry about the problem later :-) he he he. as such, the system gets bigger (and slower) and complicated over time making maintenance more tedious and complicated.

              a good designer knows when to admit if the design sucks, the earlier the better, and then goes back to the drawing board (back to step 1) and do it all over again even if it means a lot of work and delays the project. in the long run, this action is much better and makes the design more robust moving forward rather than sticking with the original design that sucks, and then, work out a solution to the problem. i have seen too many of these scenarios in the past and it never paid off to the designer and the people who supported it :-).

              going back to the event rules processing. the original design with 'no time element', as mentioned above takes a snapshot of the events and then takes an action based on the current status of the event. consider a line with points A, B, and C on it, where at each point, you get a snapshot of the event's status and then take some action based on that status. the notion 'no time element' is really not correct. an event, even in it's most basic component, without a time element (e.g. duration, delay before, or delay after) for it's input and/or output, already has a time element in it. the time element is the period between the points (e.g. A, B, C). it's there all along, it just was not considered in the original design because at that time, the concern, was more on the current status of the event.
              the snapshot of the current status, whether it's on an event level, input level, or output level, MUST ALWAYS consider the previous state of the status. this was the one that was missing in the processing logic and the design of the tables/fields to support that logic :-).

              processing should be much faster with the new basic design of the events processing as succeeding snapshots will require less task to perform since the logic is now taking into consideration the previous status in determining the current status and the action to be taken. the logic for all the levels (event level, input level, output level) in the event processing will now follow the same basic logic which will remove the complication that was done in the old design to incorporate the 3 functionalities mentioned above ... cheers

              Boo!

              Comment


              • #12
                Hi Boo,

                "it looks like most of the parts available here are all through-hole electronic parts."

                For hard to find parts, you can contact them. They have filipino agents you can talk to and you can even request sample units for free (hopefully).

                http://www.wingstech.com.ph/
                http://ph.rs-online.com/web/
                http://ph.element14.com/

                Comment


                • Boo-Semi-Retired
                  Boo-Semi-Retired commented
                  Editing a comment
                  thanks for sharing this info. i have ordered a few times with element14 before in SG as their shop is quite near my place before. i will surely check the other sites you mentioned ... cheers

                  Boo!

              • #13
                re-post as a reply instead of a comment ...

                just an update on this project of mine ... the problem (handling a huge number of events/rules for a base configuration with a max of 1350 sensors) is still unresolved as of now. the problem is basically due to the limited memory/eprom space in the MCU plus the fact that users must have a freehand (based on design) in setting up their events/rules so long as they stick to the system limit - 1 event can have a max of 8 input conditions (rules) and a max of 8 output actions (rules).

                i have explored and tested a number of possible software options to resolve the problem but the results are not really that satisfactory, even for my taste. so it seems that the only option left for me is to look at a major change in the design strategy - i was trying to avoid this as it will involve a lot of changes to the HAS system.

                the original design was to distribute the process throughout the HAS network to ensure quick response time - end-nodes to handle the sensor reading, conversion, and actual actions to events ; mid-host nodes to handle event/rules processing (this is where the space problem exist) ; main-host node handles the consolidation, batch / housekeeping process, and initiation of actions.

                after weeks of reviewing and analyzing the process and the existing programs, the best approach to take are:
                (1) also distribute the event/rules processing across all the nodes (MCU),
                - if both monitor and action sensors for an event/rule is controlled within a particular end-node, then handle it within the end-node process.
                - if the monitor and action sensors for a particular event/rule are controlled by different end-nodes which is controlled by a single mid-host node, then handle it within the mid-host node process.
                - if the monitor and action sensors for a particular event/rule are controlled by different end-nodes controlled by different mid-host nodes, then handle it within the main-host node process.
                (2) add a new validation routine in the processing of the parameter setup text file and compute that the amount of memory/eprom space required for all the files/tables required by HAS does not exceed the limit of the individual node hardware in the HAS network.

                next step now is to document all the changes that needs to be done to each of the HAS programs/functions/libraries and the changes to the system global parameters. once completed and verified to be correct, then actual revisions to the system can be started ... end result, a lot of sleepless nights for the next month or so :-( ... cheers

                Boo!

                Comment


                • #14
                  Event/Rules Split Processing ... Part 1: Node Table Design and Relationships

                  the original HAS design has the following general characteristics:

                  1. end-host node table size = max (15) sensor capacity per end-host node.
                  2. mid-host node table size = max (15*15=225) sensor capacity per mid-host node. system max capacity=1350.
                  3. end-host node only performs sensor reading and pass-on the reading to the mid-host for event/rules processing. it also performs sensor action execution based on the command it receives from the mid-host node.
                  4. mid-host node performs the event/rules processing on the sensor reading/s it receives from all the end-host nodes that it has control. after it process the reading/s, it transmits the reading to the main-host node for logging and other MIS.

                  the HAS software works based on the testing i have done.

                  but as you can infer from the above, it will only work up to a certain point. as expected from the design, once the number of event/rules conditions is quite high, due to the limited amount of MCU eprom (EPROM Erasable Programmable Read Only Memory) space, problem will occur most, likely at the mid-host node level.

                  a node's MCU normally has about 512KB of eprom. it's where the HAS configuration file of each node is stored. when each of the board is powered on, the program reads the configuration file and uses it to setup all the parameters and tables required by each of the nodes for HAS to run. so, with a fix table size design, there is only a limited amount of space remaining in the 512KB eprom for the event/rules condition which is also needed for HAS to work. in a nut shell, the problem happens when a node's configuration file size is bigger than the available MCU eprom space.

                  identifying where the problem is in the design is simple. i have reviewed the existing system design so many times and the big space problem of a node's configuration file is due to the following:
                  1. to much unused space is wasted in the sensor table definition required by HAS.
                  2. event/rules processing that also requires a lot of table definition is only done at the mid-host node.

                  coming up with a solution to the problem is a bit tricky since the embedded HAS system uses record pointers to randomly access records in a number of tables. in addition, the solution must be simple enough to avoid complexity in the program logic to keep it as small as possible.

                  overall, the solution i'm looking at is to enhance the HAS functionality with:
                  1. revise the table setup process to only contain entries for sensors that has been defined in the parameter setup. this will greatly reduce the size of the table area in the configuration file.
                  2. revise the parameter setup process to include a number of routines to compute the total space required of each configuration file for each of the nodes defined in HAS and to validate this with the MCU hardware eprom capacity to ensure that it will fit it.
                  3. provide an option to do event/rules split processing instead of just doing it in a mid-host node. by this i mean:
                  a. if an event has all it's input rules and output rules in 1 end-host node, then it's going to be handled in the end-host level.
                  b. if an event has all it's input rules and output rules in different end-host node that are controlled by 1 mid-host node, then it's going to be handled in the mid-host level.
                  c. if an event has all it's input rules and output rules in different end-host node that are also controlled by different mid-host node, then it's going to be handled in the main-host level.

                  now that there is a game plan of the changes to be done, the next posting will focus on some of the issues i have identified while documenting the detailed specs of the changes ... cheers

                  Boo!
                  Last edited by Boo-Semi-Retired; 10-09-2016, 04:54 AM.

                  Comment


                  • #15

                    Event/Rules Split Processing ... Part 2: Node Table Design and Relationships

                    one big issue i identified while documenting the logic for the changes is how to keep the integrity of the record pointers of the related tables if the number of records (table size) in each of the tables is a variable and the record position of the data, in each of the table is also a variable, for all the HAS nodes.

                    to add more complexity, the table definition and logic for each type of the nodes (main-host, mid-host, end-host) should be the same to ensure that the sub-routines are reusable in all of the 3 node programs and to keep the program's processing logic the same as much as possible.

                    if the table size is fix, it's easy, you just use a common info (e.g. a sensor's unique address) and use it as a record pointer. this was how it was done in the original HAS design. but as you can see, a lot of valuable memory space is wasted if max capacity is not use in the HAS system.
                    example: an end-host node has a max of 15 sensors that can be connected to it. so, a sensor table can be defined with a capacity of 15 entries. if the node only has 4 sensors connected to it, then the space for the 11 entries is a waste as it's not use but its still defined in the system. the same is true for the mid-host node and the wasted memory will be much bigger.

                    relationship diagram of the sensors in a table for a mid-host and its end-host nodes:

                    mid-host-1 end-host-1 end-host-2 end-host-3
                    snsr-1@port 01 [1] snsr-1@port 01 [1] snsr-1@port 07 [1] snsr-1@port 01
                    snsr-2@port 09 [2] snsr-2@port 09 [2] snsr-2@port 08 [2] snsr-2@port 05
                    snsr-3@port 10 [3] snsr-3@port 10 [3] snsr-3@port 09 [3] snsr-3@port 06
                    snsr-4@port 11 [4] snsr-4@port 11 [4] snsr-4@port 10 [4] snsr-4@port 09
                    snsr-5@port 12 [5] snsr-5@port 12 [5] snsr-5@port 15
                    snsr-6@port 13 [6] snsr-6@port 13 [6] snsr-6@port 16
                    [7] snsr-7@port 17
                    snsr-1@port 07 [8] snsr-8@port 18
                    snsr-2@port 08
                    snsr-3@port 09
                    snsr-4@port 10

                    snsr-1@port 01
                    snsr-2@port 05
                    snsr-3@port 06
                    snsr-4@port 09
                    snsr-5@port 15
                    snsr-6@port 16
                    snsr-7@port 17
                    snsr-8@port 18

                    to help visualize the issue, consider the simple diagram above and note the following:

                    1. you have 1 mid-node host which monitor and control 3 end-host nodes.
                    example: mid-host-1 controlling end-host-1, end-host-2, and end-host-3.

                    2. the sensors are connected to each end-host node depending on the user's requirement. in the sample diagram, 6 sensors for end-host-1, 4 for end-host-2 and 8 for end-host-3.
                    example: end-host-1 sensor-1 is connected to HW port 01 while for end-host-3, sensor-1 is also connected to HW port 01. both sensors can be the same (e.g. a relay switch) or both can be 2 different sensors (e.g. one is a relay switch and the other is an LED light). what is true is that they are 2 physical sensors connected to different end-host-nodes.

                    3. for mid-host-1 to monitor and control the 3 end-host nodes, it must have a copy of all the sensor readings from the 3 end-host nodes in its own memory. this is done when the end-host node transmit the reading to the mid-host whenever it performs getting a new sensor reading.
                    for getting a reading, the flow of info is always from sensor to end-host to mid-host.
                    for initiating an action to a sensor, the flow of info is always from mid-host to end-host to the sensor if the event is controlled by the mid-host. if the event is controlled by the end-host, then the flow is from end-host to sensor.
                    example: end-host-3 performs a sensor reading every 30 secs. it access the HW port of each sensor. for a monitor sensor, it gets a new reading. for an action sensor, it gets the status of the sensor. the end-host stores the info in its own table and also transmit a message to the mid-host for it to update its own table.

                    4. the issue lies in the exchange of the info between the nodes (end-host and mid-host) and the fact that the info is stored in different location which each node is not aware off - transmitting node does not know in which area of the receiving node is its info being stored.

                    one solution is for the sender to transmit additional info (e.g. end-host, sensor, port) together with the reading info to the receiver. the receiver can then sequentially read all the records in its table and find a match to know where to store the new reading info.
                    the solution will work, but the processing time will be extremely long, thus response time will be very slow - reading a whole table just to save 1 info will take a longer time to complete and you need to repeat this for each of the sensor reading.

                    a better approach is to provide the specific location of the record @ the receiving node so that the receiver can just randomly access the record in the table and update the information.
                    so how can the sender know the location of the record in the receiver's table whose number of entries (elements) is variable in nature? the real issue at hand :-).

                    one simple and old programming technique i learned in maintaining a plant automation system is the use of an offset (an area before the start of where the actual data is) value. it's basically the same way how an index file works without the key-field. actually for this case, the key-field will be the record's entry number in a table stored in memory.

                    to better understand the concept, i'll explain it using the simple diagram above:

                    1. let us use as an example the sensor-3 record of end-host-3 which is connected to port 6. this is the 3rd entry in a table of end-host-3.

                    2. looking at the mid-host-1 table, you notice that all the end-host-3 records are stored from entry [11] to [18].

                    3. so, using an offset value = entry [10] for the end-host-3 node, the node will be able to know where it's data is stored in the mid-host-1 table. note that each end-host node uses a different offset value, for end-host-1 its entry [0], for end-host-2, its going to be entry [6].
                    example: offset value + end-host table entry position = position in mid-host table

                    4. item #3 pertains to the logic of knowing the table info of a mid-host if an end-node needs to transmit a message to it.
                    if it's the reverse (e.g. mid-host transmit a message to an end-host to initiate an action sensor), then the logic to know where the sensor data is stored in the end-host node is to deduct the offset value from the mid-host table entry.
                    example: mid-host table entry position - offset value = position in end-host table

                    5. the same technique can also be applied to the other types of tables that you need to setup in the HAS network.

                    6. as in the index file concept, for the above to work to randomly access records in a table in different nodes, loading of the records or entries into a table for all the HAS nodes must follow a common sorting order that explains the relationship of the tables.

                    cheers

                    Boo!

                    Comment

                    Related Articles

                    Collapse

                    Collapse
                    • Filter
                    • Time
                    • Show
                    • New Topics
                    Clear All
                    new posts
                    No content found
                    Show More
                    • Filter
                    • Time
                    • Show
                    • New Topics
                    Clear All
                    new posts
                    Please log in to your account to view your subscribed posts.

                    Related Articles

                    Collapse

                    Working...
                    X