1
0
-1

Or DSP - Service (assuming you are using the DSP component subtypes which have no database update functions). 

So, many many years ago we built our client/server app using uniface lists (and lists of lists, all the way down) to pass the data from the server to the client. 

 I have tried in the past to switch this to XML transfers with limited success, and am re-trying again to refactor our C/S transfers.

I'd like to be able to do without DTDs as these are (from my perspective) just another thing which will prevent the transfer of all the data from client to server unless the programmer is very very careful when adding new fields to an entity that they have found all the DTDs using that entity and included them. When you don't then any existing data is WIPED from the database when being re-saved (may only be true of down entities).  So the DTD is a dangerous hole in the development. Also managing the maps is a pain in the proverbial. 

I am therefore trying to use componenttostruct, structtojson, and vice versa to pass the data. I am using the /reconnecttags but can't see any sign of them in the JSON, (I don't think). 


Short version of the question, how are people generally passing complex data from server to client and vice versa, maintaining the modification status? If it's XML are there any tricks to maintaining DTDs which make them less of a trap? 

Cheers. 

Iain

    CommentAdd your comment...

    6 answers

    1.  
      1
      0
      -1

      So, after all this mucking about, the answer (it appears) is to declare the struct parameters as byVal in the operation. There is then no need for the XML or Json transforms, or flattening the tags. 

      At that point, 

      componenttostruct/reconnectflags/firetriggers v_struct, "entityname" 

      can pass its data to the client. 

      In the client, I use the procedures below (global procs to minimise compiled code). To 'map' the component subtypes together.  The component variables are structs (for speed).

      1. Setup mappings (cp_mapssvtofrm called as cp_map_ssvtofrm( "",$frmmapping$). Mapping needs to be INOUT or else it wipes it leaving the top recursion, not sure why. 

        params
        	string p_entity : IN
        	struct p_mapping : INOUT
        endparams
        
        variables
        	string v_frm_ent, v_ssv_ent, v_inners, v_sub, v_output
        endvariables
        
        if(p_entity = "")
        	v_inners = $componentinfo($instancename,"OUTERENTITIES")
        else
        	v_inners = $entinfo(p_entity,"INNER")
        endif
        forlist v_frm_ent in v_inners
        	if($scan(v_frm_ent,".") > 0)
        		v_frm_ent = v_frm_ent[1,$scan(v_frm_ent,".")-1]
        	endif
        	v_ssv_ent = v_frm_ent[1,$length(v_frm_ent)-3]
        	v_ssv_ent = $concat(v_ssv_ent,"SSV")
        	p_mapping->"%%v_SSV_ent%%%" = v_FRM_ent
        	if($entinfo(v_frm_ent,"INNER") != "")
        		call cp_map_ssvtofrm(v_frm_ent, p_mapping)
        	endif
        endfor
        return 0
      2. Post load to form, change struct data before structocomponent. (This one mostly nicked from Daniel's code.) (cp_mapentstruct called as cp_mapentstruct(v_struct, $frmmapping$). Mapping doesn't need to be INOUT as the struct is only read. 

        params
        	struct p_struct      : IN
        	struct p_mapping : IN
        endparams
        
        variables
        	numeric i, n
        	string v_name
        endvariables
        
        ;If this struct has more than 1 member then loop over them
        ;calling cp_mapentstruct recursively on each
        if (p_struct->$collSize > 1)
        	
        	n = p_struct->$collSize
        	for i = 1 to n
        		if (p_struct{i}->$tags->u_type != "field")
        			call cp_mapentstruct(p_struct{i}, p_mapping)
        		endif
        	endfor
        	
        	return 0
        	
        endif
        
        ;Now see what type it is, component, entity, occurrence
        selectcase p_struct->$tags->u_type
        	case "component"
        		
        		;This is the component node, so call this function
        		;recursively on all the entity nodes under it
        		call cp_mapentstruct(p_struct->*, p_mapping)
        		
        	case "entity"
        		
        		;This is an entity node, so call this function
        		;recursively on all the occurrence nodes
        		;under it (If they exist)
        		if (p_struct->$membercount > 0)
        			call cp_mapentstruct(p_struct->*, p_mapping)
        		endif
        		v_name = p_struct->$name
        		v_name = v_name[1,$scan(v_name,".")-1]
        		if(p_mapping->"%%v_name%%%" != "")
        			p_struct->$name = $replace(p_struct->$name,1,v_name,p_mapping->"%%v_name%%%",-1)
        		endif
        		;            if (p_struct->$name != "")
        		;                p_struct->$name = $item(p_struct->$name, "<myStructMapping>")
        		;            else
        		;                p_struct->$name = $valuepart($itemnr(1, "<myStructMapping>"))
        		;            endif
        		
        	case "occurrence"
        		
        		;This is an occurrence node, so just call this function
        		;recursively on all the entity or field nodes under it.
        		call cp_mapentstruct(p_struct->*, p_mapping)
        		
        endselectcase
        
        return 0
        
        end ;cp_mapentstruct

      There's some faffing about in here, as OUTERENTITIES gives the entname.model, but INNER only gives the entname, and there's no way (AFAIK) to get the model from the entity name. So we have to strip the model from the outers to make them consistent, and then strip the model from the name of the struct node to compare to the mapping. But it IS self-maintaining based on the painted entities and doesn't require developer input. 


        CommentAdd your comment...
      1.  
        1
        0
        -1

        Here's some code for doing the sub-type mapping:

        ;----------------------------------------------------------------
        entry doMapping
        ;----------------------------------------------------------------
        	params
        		struct p_struct      : IN
        	endparams
        	
        	variables
        		numeric i, n
        	endvariables
        	
        	;If this struct has more than 1 member then loop over them
        	;calling doMapping recursively on each
        	if (p_struct->$collSize > 1)
        		
        		n = p_struct->$collSize
        		for i = 1 to n
        			if (p_struct{i}->$tags->u_type != "field")
        				call doMapping(p_struct{i})
        			endif
        		endfor
        		
        		return 0
        		
        	endif
        	
        	;Now see what type it is, component, entity, occurrence
        	selectcase p_struct->$tags->u_type
        		case "component"
        			
        			;This is the component node, so call this function
        			;recursively on all the entity nodes under it
        			call doMapping(p_struct->*)
        			
        		case "entity"
        			
        			;This is an entity node, so call this function
        			;recursively on all the occurrence nodes
        			;under it (If they exist)
        			if (p_struct->$membercount > 0)
        				call doMapping(p_struct->*)
        			endif
        			if (p_struct->$name != "")
        				p_struct->$name = $item(p_struct->$name, "<myStructMapping>")
        			else
        				p_struct->$name = $valuepart($itemnr(1, "<myStructMapping>"))
        			endif
        			
        		case "occurrence"
        			
        			;This is an occurrence node, so just call this function
        			;recursively on all the entity or field nodes under it.
        			call doMapping(p_struct->*)
        			
        	endselectcase
        	
        	return 0
        	
        end ;doMapping
        
        

        The required define for the mapping could look like this (this would be for a SSV component:

        #define  myStructMapping = ENT1FRM.MODEL1=ENT1SSV.MODEL1·;ENT2FRM.MODEL1=ENT2SSV.MODEL1
        
        

        No rocket science and done in about 15 minutes...


          CommentAdd your comment...
        1.  
          1
          0
          -1

          Here's the code for moving/restoring the reconnect info for XML:


          ;----------------------------------------------------------------
          entry moveReconnectData
          ;----------------------------------------------------------------
          params
             struct p_struct      : IN
          endparams
          
          variables
             numeric i
             numeric n
          endvariables
          
             ;If this struct has more than 1 member then loop over them
             ;calling moveReconnectData recursively on each
             if (p_struct->$collSize > 1)
          
                 n = p_struct->$collSize
                 for i = 1 to n
                    call moveReconnectData(p_struct{i})
                 endfor
          
                 return 0
          
             endif
          
             ;This struct only has 1 member so process it.
             ;First, move the u_type tag into an attribute
             p_struct->$tags->u_type->$parent = p_struct->$tags->$parent
             p_struct->u_type->$index = 1
             p_struct->*{1}->$tags->xmlClass = "attribute"
          
             ;Now see what type it is, component, entity, occurrence
             selectcase p_struct->u_type
                 case "component"
          
                    ;This is the component node, so call this function
                    ;recursively on all the entity nodes under it
                    call moveReconnectData(p_struct->*)
          
                 case "entity"
          
                    ;This is an entity node, so call this function
                    ;recursively on all the occurrence nodes
                    ;under it (If they exist)
                    if (p_struct->$membercount > 0)
                       call moveReconnectData(p_struct->*)
                    endif
          
                 case "occurrence"
          
                    ;This is an occurrence node, so just call this function
                    ;recursively on all the entity or field nodes under it.
                    call moveReconnectData(p_struct->*)
          
                    ;Move reconnect data into attributes
                    p_struct->$tags->u_status->$parent = p_struct->$tags->$parent
                    p_struct->u_status->$index = 1
                    p_struct->*{1}->$tags->xmlClass = "attribute"
                    p_struct->$tags->u_crc->$parent = p_struct->$tags->$parent
                    p_struct->u_crc->$index = 1
                    p_struct->*{1}->$tags->xmlClass = "attribute"
                    p_struct->$tags->u_id->$parent = p_struct->$tags->$parent
                    p_struct->u_id->$index = 1
                    p_struct->*{1}->$tags->xmlClass = "attribute"
          
                 case "field"
          
                    ;Fields will always be leaves, so no need to do anything here
          
             endselectcase
          
             return 0
          
          end ;moveReconnectData
          
          
          ;----------------------------------------------------------------
          entry restoreReconnectData
          ;----------------------------------------------------------------
          params
             struct p_struct      : IN
          endparams
          
          variables
             numeric i
             numeric n
          endvariables
          
             ;Clear out tags added by xmltostruct
             p_struct->$tags->*->$parent = ""
          
             ;If this struct has more than 1 member then loop over them
             ;calling restoreReconnectData recursively on each
             if (p_struct->$collSize > 1)
          
                 n = p_struct->$collSize
                 for i = 1 to n
                    call restoreReconnectData(p_struct{i})
                 endfor
          
                 return 0
          
             endif
          
             ;This struct only has 1 member so process it.First see what
             ;type it is, component, entity, occurrence
             p_struct->"u_type"->$tags->*->$parent = ""
             p_struct->"u_type"->$parent = p_struct->$tags
          
             selectcase p_struct->$tags->u_type
                 case "component"
          
                    ;This is the component node, so call this function
                    ;recursively on all the entity nodes under it
                    call restoreReconnectData(p_struct->*)
          
                 case "entity"
          
                    ;This is an entity node, so call this function
                    ;recursively on all the occurrence nodes
                    ;under it (If they exist)
                    if (p_struct->$membercount > 0)
                       call restoreReconnectData(p_struct->*)
                    endif
          
                 case "occurrence"
          
                    ;This is an occurrence node, so just call this function
                    ;recursively on all the entity or field nodes under it.
                    call restoreReconnectData(p_struct->*)
          
                    ;Move reconnect data into tags
                    p_struct->u_status->$tags->*->$parent = ""
                    p_struct->u_status->$parent = p_struct->$tags
                    p_struct->$tags->u_status->$index = 2
          
                    p_struct->u_crc->$tags->*->$parent = ""
                    p_struct->u_crc->$parent = p_struct->$tags
                    p_struct->$tags->u_crc->$index = 2
          
                    p_struct->u_id->$tags->*->$parent = ""
                    p_struct->u_id->$parent = p_struct->$tags
                    p_struct->$tags->u_id->$index = 2
          
                 case "field"
          
                    ;Fields will always be leaves, so no need to do anything here
          
             endselectcase
          
             return 0
          
          end ;restoreReconnectData
            CommentAdd your comment...
          1.  
            1
            0
            -1

            So I've tried structtojson and structoxml and there is no sign of the reconnecttags in the strings they output. As such the retrieve/reconnect fails to reset the modification status. 

            Much as I'd like to say I added fire to this discussion, it does seem I'm screaming in the wind here. I can only presume that no-one is actually using Uniface in the method they suggest as it's just broken... 

            Iain

            1. Iain Sharp

              I can see the $tags in the created struct, But they are not included in the XML (or JSON), there doesn't appear to be a method for including them. 

              I am wondering if I can do a pass through the struct, copying the contents of a node's $tags, to a non tag struct, then converting then changing them back when using the struct at the other end. This is getting very complex though. Although, done properly, it would be reusable, flexible code which would still cope with new fields, so still better than using DTDs. 

              All in all, I do feel like I'm beating my head against a wall... 

              Iain

            2. Daniel Iseli

              This is not really broken, but rather works by design.

              Before raising your blood pressure, please continue reading...

              When you do componentToStruct/reconnectTags and then structToComponent then this work without any further changes.

              If you, however, convert the Struct to either XML or JSON then you need (as it works now) to move the reconnect tags before you can do the conversion. For XML you would create XML attributes that hold the reconnect info. For JSON you need to do something similar. On the way back you need to reverse the process. Granted, this requires a bit of work, but it's possible to create a generic routine (one for XML and one for JSON) that does this.

              I'm sure that I have a possible sample routine somewhere lying around. I'll have a look when I get some spare time.

              And in the meantime you could have a look at (e.g.) this Struct example:

              > Scripting Application Behavior > ProcScript Syntax > Structs > Example: Looping Over Structs

              I've already used this numerous times if I needed to manipulate a Struct in a generic way.

              I hope this helps.

              Daniel

            3. Iain Sharp

              Okay, it's not broken against the design. But the design is broken. (smile) 

              What I am looking for is a simple, repeatable way to design an interface from the application server to the client, to pass complex datasets including their modification statuses etc., preferably in less than (say 3 ) lines of code. Which works in every component, without the developer having to manually maintain lists of fields or lose data. 

              So xmlsave is broken because DTD's are (almost by definition) out of date.

              componenttostruct is broken because the system doesn't pass structs between the application server and the client (which it could do, using XML, JSON or something else). 

              componenttostruct/structtoXXXX is broken because the tags are not included (either automatically or via a switch) in either the xml or json. 

              I agree that if I faff about and write code to pass through the struct and 'de-tag' the tags, I can get it to work and that once I've written that code (which I spent all afternoon doing and now works, an example to follow) I can re-use that. 

              My point is, and it's one I've made to Uli a lot, is that WE SHOULDN'T HAVE TO. I would like uniface to post the definitive method of passing data from client to server and vice versa whilst maintaining the modification statuses, and if it's xmlsave/xmlload, then I'm telling you that silently wiping my data because I forgot to include it in the DTD is broken..... (smile) 

              A change of structtoXXXX/reconnectags or even just including the reconnecttags from the struct in the JSON/XML because they are there, (and why would they be there if you didn't want them?) would make it not so broken. 

              It's been what, 10 years no since we got retrieve/reconnect? Isn't it time it was useful instead of the usual, hack it till it works? 

              BTW componenttostruct v_struct, "ENTITYNAME" does not provide decent output for structtocomponent without hacking that about as well. Also, no simple mapping, so how do you go from FRM component subtypes to SSV component subtypes and vice versa. 

              Regards, 

              Iain

            4. Iain Sharp

              My versions of the tags routines. (Which are proof against the uniface adding or changing the tags names/contents. )

              ; $tags to struct nodes. 
              entry lp_flatten_tags
              	params
              		struct p_struct : IN
              	endparams
              	variables
              		numeric v_counter
              		string v_name
              	endvariables
              	
              	if(p_struct->$tags->$istags & p_struct->$tags->$membercount > 0 )
              		p_struct->keep_tags = $newstruct
              		for v_counter = 1 to p_struct->$tags->$membercount
              			v_name = p_struct->$tags->*{v_counter}->$name
              			p_struct->keep_tags->"%%v_name%%%" = p_struct->$tags->*{v_counter}
              		endfor
              	endif
              	for v_counter = 1 to p_struct->$membercount
              		if(p_struct->*{v_counter}->$name != "keep_tags")
              			call lp_flatten_tags(p_struct->*{v_counter})
              		endif
              	endfor
              	
              	return(0)
              end

              and

              ; struct nodes back to $tags.
              entry lp_replace_tags
              	params
              		struct p_struct : IN
              	endparams    
              	variables
              		numeric v_counter
              		string v_name
              	endvariables
              	
              	if(p_struct->keep_tags->$membercount > 0)
              		for v_counter = 1 to p_struct->keep_tags->$membercount
              			v_name = p_struct->keep_tags->*{v_counter}->$name
              			p_struct->$tags->"%%v_name%%%" = p_struct->keep_tags->*{v_counter}
              		endfor
              		p_struct->keep_tags->$parent = ""
              	endif
              	for v_counter = 1 to p_struct->$membercount
              		if(p_struct->*{v_counter}->$membercount > 0)
              			call lp_replace_tags(p_struct->*{v_counter})
              		endif
              	endfor
              	return(0)
              end



            5. Iain Sharp

              Oh, and while I'm here, why do I have to know the implementation layout before I can code something which just works? 

              If I am running all the services in the client, without an application server, then I can pass the struct as a parameter. 

              If I move the services to the application server as an implementation decision, then whilst this will compile without issue, it will fail at implementation because you can't pass a struct across the divide. 

              Why doesn't the parameter handler within Uniface work out whether the struct needs to be converted, and convert it itself into something whcih doesn't lose information. I appreciate that some function would be lost, (changing the struct in one not changing it in the other) but it's light years better than just failing.... 

              Still cheerful, 

              Iain

            6. Daniel Iseli

              Are your reading glasses broken (as well)? (wink)

              > Scripting Application Behavior > ProcScript Syntax > Structs > Struct Parameters

              "Structs are passed by value when struct parameters are declared in a public operation, or when the byVal qualifier is used in the declaration."

              "Passing Structs by value is the only way to exchange information between components that are running in different processes, but it is fundamentally different from passing Structs by reference, because a copy of the Struct is made."

              And yes, you need to be aware of the consequences when running services remotely. It is of course advisable to use ByVal wherever possible, since it's faster (because just a memory pointer is passed and not a copy of the Struct).

              I agree that there (still) is room for improvement how Uniface "handles" Struct's. But this does not mean that something is broken. It's just working as advertised. Maybe your expectations are broken? (tongue)

              No hard feelings and stay cheerful. (smile)

              Daniel

            7. Iain Sharp

              Hmm, I’m only used to public and partner in operations in web stuff. Not come across it in ‘normal’ operations. 

              If that works, I can get rid of all this rubbish, and it turns out that is what I am looking for, and all the rest of this faff above is without merit. 

              I am still surprised there isn‘t just an example somewhere of how one SHOULD do intercomponent communication, so we can work on it, maybe we should GIT an example together and crowd source some standards.... 

            8. Iain Sharp

              So, simply declaring the operation as public does not, as the documentation would suggest, allow passing byVal by default, the declaration has to be specific. No (real) issue there. 

              However, in attempting to write a routine to build the mapping automatically, I have encountered a lack of information. 

              $entinfo(XXX, "SUPERTYPE") returns the ultimate supertype of the component subtype. Is there possibly a hidden function (or compile-time constant even) which returns the PARENT value from ucgroup? So If I have multiple functional subtypes,  painted in the same component (as component subtypes of the functional subtype) then I can work out the functional subtype name? Otherwise using $entinfo will return me the same mapping for all (in this case 7) uses of the same table. 

              I know you are manually maintaining the mapping, and I can think of a way to manually put it in the collection operations of each entity in the model, and then build that from a loop through outerentities etc, 

              Or I can manually maintain information the system should just know in every component I want to use it in.... 





            CommentAdd your comment...
          2.  
            1
            0
            -1

            Hi guys,

            Thanks for start up and giving fire to this subject!

            Gianni

              CommentAdd your comment...
            1.  
              1
              0
              -1

              Hi Iain,

              In a C/S - what would be the benefit in doing this?  Locking is gone, update concurrency is gone (unless you implement $occcrc everywhere), unless you're very careful stepped hitlists are gone - I could go on...

              XML is expensive - I totally get your point re DTD's - however, could you distribute the DTD 'source' from DICT with your application and store in a runtime db table - with a $DBMS_FIRST setting?  Everything comes at a cost I guess..

              I guess Uniface sales must love you due to the Uniface Application servers you'd need to purchase on the way through as well...  (wink)

              Only saying this because we're running 4500 users across some 300+ office servers - to a central, umpteen TB Oracle 12 server.  Just the mere thought of running server based data retrival makes me shiver...

              Regards,

              Knut


              1. Iain Sharp

                Well, we already do this, we just do it using putlistitems/getlistitems, which does NOT carry the modification status of fields or occurrences. 

                The XML functions (xmlload/xmlsave) and retrieve/reconnect do (mostly) allow for this and are the same therefore as optimistic locking. 

                We are therefore already licensed for application servers. 

                We'd have to rewrite the whole system to connect the UI direct to the database I was looking to make the current data set passing more efficient and functional. 

                Iain

              2. Iain Sharp

                All this because when we were looking at languages > 15years ago Uniface was sold as an N-tier object oriented development language. What we have now is the 15year old result of us trying to beat uniface 7/8 into something resembling either OO or N-tier... 

              3. Knut Dybendahl

                I presume you added $occcrc to the entities as a non-dbms field, and include that field in all back'n forth between client and server?  At the time of reconnect - rather than having to compare each field - the crc check would be much faster to do...


              4. Knut Dybendahl

                and as for the dtd mapping - if it's stored in the db at least you don't have to update the dol/urr whenever the dtd changes...

              5. Iain Sharp

                Not sure $occcrc existed when we built the original system. 

                Using xmlsave/xmload correctly with retrieve/reconnect is SUPPOSED to do all that for you within the uniface code. I do believe it works with crc values within the XML as well as whether the client THINKS it's modified the data. 

                componenttostruct/reconnecttags is supposed to do the same thing within structs. But then you can't pass a struct from one machine to another so you have to do structtojson (or structtoxml) to give you something which will pass. I was looking to see what the current approved 'standard' should be. 

              6. Iain Sharp

                We stopped using dol/urr a long time ago, We're all about the uar now. (smile) Way better for patching. The DTD is, therefore, a separate file now and easy to manage on that front. It's the maintenance of the thing, every field in the entity has to be explicitly added manually. If it isn't then it's not included in the passed data. If it's not included then the reconnect wipes the data from the database entry. So two programs with two DTDs (different structures) can therefore add data and take it away, because a developer missed the update  on a DTD. 

                Oh, and it's not possible (easy) to search DTDs to do a validity check that it's not been missed. And the only thing you can be sure about with manual documentation is that it's out of date/wrong.... 


              CommentAdd your comment...