Data Exchange IT webinar – Information for IT and Technical staff
Transcript for the Data Exchange IT webinar held 20 June 2018
Hello everyone and welcome to the Data Exchange IT webinar. My name is Rose Pearson and I am with the Data Exchange training team and it is my pleasure to be taking you through this webinar session today. This presentation should take about one hour, including any questions that you may have, so please send them through and I’ll show you how to do that in a moment.
Joining me today and responding to your questions will be James who works within the Data Exchange Application team.
I would like to begin by saying that we are meeting today on many traditional lands around the country and I want to acknowledge the traditional owners of those lands and pay our respects to elders past and present.
Firstly, some reminders on how to use the webinar control panel. Throughout the webinar remember there is the control panel that should have appeared on your screen, normally on the right-hand side of the screen. Clicking on any of the grey ribbons will expand those particular areas. Remember to check your sound you’ll notice that in this picture the microphone is green, but your microphone symbol should be red and this indicates that your audio has been muted. So if it isn’t red, please ensure that if you can click that little symbol being so that it does turn red, otherwise we can hear what’s happening on the other side of the microphone. Remember that even though your audio has been muted, you can actually type your questions in the question ribbon. You’ll also notice that in the handout section there, there is a presentation slide pack of what we’re going through today so you can actually print that off at your leisure. If you can’t access that handout, please let us know and we can send that to you after the webinar.
If the control panel hasn’t been used for period of time, it automatically minimises from view. So to expand the panel again all you need to do is to click on the orange arrow and it will display or expand that control panel.
Let’s take a look at what we going to be discussing during today’s webinar. We will have a quick recap of the different upload methods that are available to organisations. We will then go over the suggested frequency for uploading and the benefits of this, and also the stages of upload. We will take a look at the reference data, we’ll explain what some of the expectations are prior to actions for uploading should be.
We will also take a look at the technical specifications that are available on the Data Exchange website and what their purposes are. We’ll have a chat about the different release cycles for the Data Exchange and we will mentioned the system enhancements that are coming up in August.
Once again there will be an opportunity to ask questions throughout the session today as well as at the end of the session for any other questions that come through.
This webinar will be recorded and placed on the Data Exchange website under the Webinar library in the coming weeks, so keep an eye out if you’d like to revisit it at all or anyone in your organisation area there can view it.
So as most of you are aware, the Data Exchange allows you to submit data in three different ways. These are system-to-system, file upload or the free web-based portal.
File upload and system-to-system are applicable for organisations wishing to use their own client management systems to meet reporting requirements. System-to-system transfer means that the organisation wishes to continue using their own client management system or CMS. System administrators or IT vendors for these CMS’s will have modified the specifications of the software so that Data Exchange relevant fields are automatically uploaded to the Data Exchange at regular intervals, such as midnight each night. Technical specifications on the coding required can be found on the Data Exchange website.
File upload is similar to system-to-system transfer except that in this case the system administrators or IT vendor’s would have created a function where someone in the organisation can export and upload their data in bulk to the Data Exchange in an XML format. The reference data codes that are required and are to be sent to your IT vendor are found in the web-based portal as well. Now for both the system-to-system and also file upload, the technical specifications are found under the IT access tab under the System-to-system transfer and Bulk upload headings respectively.
With both these methods the organisation’s IT vendor system administrator would have contacted our Helpdesk to gain access to the Data Exchange staging environment to develop and test uploads and system-to-system transmissions prior to submitting live data into the Data Exchange. Doing this will assist in reducing errors.
There is the third option that an organisation can use and it is the free web-based portal which allows organisations to directly enter and report their data. It’s good to note that no option is preferred by the department. Providers should choose the method that best suits the business needs of the organisation. It’s also important to note that an organisation can change the upload method at any time.
For example, they may want to start entering data into the web-based portal while working with their IT developers to build the system-to-system or bulk file upload capability for the longer term. If this is the case, it’s very important that all parties ensure that the client IDs are different or unique or if sequential that they start after the client IDs that have already been used and transmitted to the Data Exchange, otherwise there is the risk that the data that has already been submitted could be overwritten. The Upload methods task card on the Data Exchange website under the Training resources tab has more information on this.
Organisations should upload data to the Data Exchange at frequent intervals within reporting period; this approach can have the following benefits. There is smaller file sizes, which means faster upload times and less possible errors needing to be resolved and resubmitted. You also have on-the-fly Data Exchange reports containing entered data, which provides availability of reports and up-to-date tracking throughout a reporting period as opposed to only afterwards.
Organisation should attempt to avoid transmitting data in large quantities towards the end of the reporting period extension months, which are January and July. If upload errors occur and need resolving and resubmitting there may not be enough time to do so. Data Exchange reports may not be available to view until close to or after the reporting period extension end. If there are discrepancies or unexpected numbers in reports from incorrect import then there will be little or no opportunity to actually fix these up. Also our Helpdesk is extremely busy during this peak period, so they may not be able to answer your query or action your request prior to the deadline.
There are several different statuses of an uploaded file that indicate where it is in the process. When a file is first uploaded to the Data Exchange it will have the status of ‘pending’. Pending means that the file is ‘queued’ but has not begun processing yet. The time a file will be in pending status depends on a couple of factors. Firstly, the amount of files for the same organisation that proceeded in the queue at the time in relation to the last processing run. Only one upload file per organisation will be processed at any time. This is an intentional rule that’s been put in place to allow every organisation an equal opportunity for file processing and it negates against certain unwanted scenarios, like when one organisation with many files might hold that point in the queue over the other organisations.
It also prevents leapfrogging where files processing at the same time might overtake another file which contains data that the front running file might depend on existing in the system prior. An example of this would be a file containing Client data and another file containing Case and Session data – Cases and Sessions obviously rely on Client records being present in the Data Exchange, so if both files were running concurrently, and then the Case/Session file was processing faster, this would result in errors as Client records may not have yet been processed.
Files are processed in the order they are uploaded, so they should be uploaded in the order that they need to be processed in. It is the provider’s responsibility to sequence them in the correct and desired order when uploading.
The Data Exchange has a process that runs at frequent intervals and will take the first file in the queue for the organisation and moves it to ‘processing’. Subsequent files will only be picked up for processing once the previous file had completed processing.
The time that ‘front-of-queue’ or individual files might sit in the ‘pending’ status could depend on the distance between the previous and next processing run. Those files uploaded immediately after process run have a longer time in pending as the distance to the next run will be at its longest.
This should not influence when or how often a provider should upload a file, but rather provide some clarification about why some files might appear to be ‘pending’ longer than others.
The next status in the sequence is ‘processing’, which is the file data being read and written to Data Exchange. The duration of this stage is variable, dependant on file size, number of records and lines contained within, and number of validations needed per record or the item.
Once a file has completed processing, it will result in one of two status:
- ‘Successful status’ indicates that all records within the file met all rules and were uploaded to Data Exchange. There may be presence of an ‘exclamation mark’ icon next to this file in the Data Exchange web portal, which indicates that there are some ‘warnings’ that the uploader should be aware of. This could include things like where an address could only be validated by locality, or extra items sent for a record were not required and therefore ignored. Providers can download a file report to see which records received a warning.
- ‘Failed status’ indicates that one, or multiple records within the file failed validation and were unable to be processed. Providers can download a file report to see which records received an error, as well as the reason why. Please note – it only takes one failed record to show a ‘failed file status’. Any entity not appearing on the file error report should be assumed to have successfully uploaded. These records will not need to be resubmitted.
Multiple upload files can be ‘queued up’ if there are multiple, and once in the queue they will automatically process one after another. This can be useful if an organisation wishes to load up the queue prior to leaving for the day, and have the files process overnight.
There is a caveat to this approach in that errors in early files, that subsequent files have dependencies on, may result in a cascading effect of errors.
Therefore it is recommended that when queuing files, providers consider the number of files they wish to queue, and the files relationship to each other. This extra consideration may contain unforeseen errors to a reasonable level.
For organisations that use Bulk file upload as their upload method it is very important that they download the necessary reference data and schema files for their organisation, outlets and programs.
These files are dynamic and codes within will change to reflect value changes on the Data Exchange; it is recommended that the Reference Data file is downloaded every 3 months to refresh any new values added to existing Data Exchange fields.
Outlet Activity and Organisation Activity files will need to be downloaded each time there is a new, or a change to organisations Outlets or Program activities respectively and these need to then be sent to their IT vendor.
These files can be found in the web based portal and I am just going to show you how to get to that now; here on the screen is the Data Exchange web- based portal Homepage and the Data Exchange organisation administrator/s for the organisation will be able to access these files under the ‘My organisation’ heading and select the ‘Reference data’ hyperlink. More information can be found in the ‘Upload methods’ task which is on the Data Exchange website under the Training resources tab.
WSDL stands for Web Services Description Language. It describes the functionality for a Web Service, in this instance, Data Exchange system-to-system solution. A WSDL file is a machine-readable file, which provides a description of how the Web Service can be called, expected parameters and criteria and what data structure is returned.
IT vendors or system administrators can use the Data Exchange WSDL file to develop a system-to-system interface with the Data Exchange from their own systems. WSDL can be obtained from the links shown on the screen.
When uploading files to the Data Exchange, there are certain preparatory actions an organisation or IT vendor are expected to take to ensure valid and correct data is transmitted to the Data Exchange.
The Data exchange offers an XSD schema file, which organisations can validate their upload file against to check that data rules and requirements for each field are met. Whilst this resource would be certainly used in any initial development, providers and IT vendors are expected to continuously validate their generated files against this schema anytime they wish to upload live data to the Data Exchange.
This will ensure that any changes to the schema (and rules) have been catered for, and pick up any unintentional issues arising from the Client Management Software since initial development.
The Data Exchange offers a staging environment, which is set up to allow organisation and IT vendors a practical resource to develop against Data Exchange items, for a range of different scenarios and an environment to test file uploading and system-to-system Web Services interfaces.
When there are future changes to the Data Exchange, the Data Exchange staging will be updated as much in advance of these changes as possible so that users can begin to familiarise themselves with any changes that may affect, or be relevant to them, and practically develop and test for these as necessary.
For access to the Data Exchange staging environment, please contact the Data Exchange Helpdesk. Contact details will be available at the end of the presentation.
Remember to include in your email:
The first and last name of the person requesting the account
- A phone number and email address for the person
- The name of the Organisation or business the staging account is for.
There is an expectation that when a file is uploaded to the live Data Exchange, it has been thoroughly validated and tested prior to this. The Data Exchange live environment isn’t the place to test files or “catch” errors. Although it will certainly reject files that do not pass validation rules, these should be tested, discovered and resolved in previous exercises in the staging environment.
You can see on the screen the technical specifications. These documents specify the technical requirements including rules and data formats for uploading or transmitting data to Data Exchange. These documents are not intended to teach users how to write code or read XML, so prior knowledge of, or experience with XML or Web Services is required. The technical specifications that are on the Data Exchange website under the ‘System to system transfer’ or ‘Bulk uploads’ heading contain information and configuration for the upcoming system enhancements that will occur sometime in August as well.
Should there be changes released to the Data Exchange (for example – enhancements), they will occur in one of two release cycles that occur annually. These are planned for August and February to occur as early as possible after the end of a reporting period, which allows for almost a full reporting period with the updates. IT vendors are encouraged to keep an eye out for these releases by subscribing to the Data Exchange mailing list, to ensure they have advance notice of changes. To subscribe you go to the Data Exchange website and on the Homepage, select the Subscribe button on the right hand side.
I want to make you aware that system enhancements to the data exchange will occur in August this year. There will be changes to optional fields as well as the updating of some SCORE domains. For example there will be a change to the homeless indicator field, which is an optional data item in the extended data set.
Information on this change was sent to IT vendors on the 8th June. If you didn’t receive this information please contact our Helpdesk. Once these enhancements have been implemented and go live, we will be holding a webinar for organisations and IT vendors if you wish to attend. This webinar will demonstrate these enhancements for users of the web-based portal.
In the meantime, a System enhancements webinar has already been delivered and that discusses some of these enhancements using screen shots only. This webinar was recorded and it is available on the Data Exchange website under the Webinar library tab.
The technical specifications currently on the Data Exchange web-site do reflect the upcoming August enhancements, we are asking organisations if you are attending today to please ensure IT vendor system administrators are using these current versions in updating any of the coding etc.
More information on what we discussed can be found on the Data Exchange website. There is also our dedicated Helpdesk, which can assist you with any further questions or support or access to that staging environment. As we have mentioned too, you can subscribe to keep up to date receive any Data Exchange news.
We are going to hand over to James to answer the questions for you live.
Good afternoon everyone. Let’s get to the first question. So the first question I have here is in relation to client consent indicator when using system-to-system transfer. Essentially the consent reflects the same information across all three methods of transmission. Where we ask that question is for providers to say that they are comfortable for that information being transferred to the Data Exchange. So the question that is asked to the client is asked by the provider on behalf of the Data Exchange. System-to-system users and bulk file upload users are not required to gain consent from their clients and have the option to send through an SLK instead of a given name and a family name. So where consent is not given or not asked it should be indicated as ‘False’ and an SLK generated before upload and that sent to us instead of the family name and the given name.
The next question is, ‘if there is a failed upload do any of the records in the file get uploaded or does the entire file fail and need to be uploaded later?’ I think this one might have been answered in the presentation.
No, if the file contains say 100 records and there is one failure, the file will read as ‘failed’. If the 99 successful records are not mentioned in the report it should be assumed that they have gone through successfully, so it will only take one failed record to return the status of ‘failed’. But those records that met validation and met all the rules will successfully upload. We would encourage that you do verify this, check a couple of random records just to make sure, but the standard practise is that unless it is showing on the error report as a failure then it can be assumed that it uploaded. Providers then have one or two options to resolve this. They can target that ‘failed’ record or records and return those in the subsequent upload or they can send the whole upload file again with the ‘failed’ records corrected. Now provided that no IDs have changed in the interim, providers won’t see any issues regarding the records that are already on, the replacement IDs will recognise that the record already exists on the Data Exchange and not update it; but obviously those records that have failed will not have an ID or be present in the Data Exchange so they will be written provided that all errors have been corrected.
The next question is ‘is there a document that exists of errors and explanations for system-to-system transfer?’ No there is not. Errors and what the errors are will come back in a response when a system‑to-system transfer is attempted. We have looked into building a document such as this but there are a few reasons why we don’t know if it will be that useful. A lot of the issue has to do with the person actually submitting the data and the person who built the configuration.
It’s often more than likely not the same person, so an error and explanation of that error to one person might not necessarily mean the same thing to the person that developed the interface with the CMS. Third-party proprietor administrators also have the ability to translate DEX errors into something more specific for the organisation. An error that we can demonstrate and explain where it might be able to be resolved might read completely different to a provider. It’s something that we will still investigate and if we get more feedback that that would be a resource that would be useful, then it is something that we will definitely consider and look into publishing.
The next question is ‘Can you confirm during the extension period for example January and July that the following period will not be opened?’ So January and July are our extension periods per year and that’s to finalise any data that occurred in the previous last six months. Anything that goes past those dates for the reporting period is actually considered a system reopen. So the extension period is the month of January and the month of July and as a rule dates past those deadlines will not be opened unless under exceptional circumstances.
Terrific question there on the reporting period. I would like to just let you know that we will have an end of reporting period webinar coming up on 27th June, so just later this month. If you would like to register to attend that webinar, please go to the Data Exchange website under the find training tab and you can register to attend their thank you for that question and we will hand back to James.
The next question is for the case end date which is a new DEX field. ‘Need to understand that no further data can be submitted if an end date is set including past activity. Is this correct?’ It depends on the end date specified. For example, if the end date of the case was 30th March then anything between the 1st of January and the 30th March can be added against that case even after the end date is added. It’s not that putting an end date in that case makes it immediately unavailable to add any sessions or SCOREs against, provided that any sessions or SCOREs were conducted on a date that is prior to what the case end date is.
That’s one of the system enhancements that’s coming up in August. Again we will have a system enhancement webinar when that becomes live, but we do have one that has already been recorded using the screen shots of what we expect to see in the Data Exchange when those enhancements come into place. You can find that webinar recording under the Webinar library tab.
This one here is a two-part question. ‘Some programs have been advised that previously voluntary partnership data is going to be considered mandatory in upcoming contracts and testing shows that partnership data fields do not flag missing data when using a system-to-system upload.’ Any concept or theory around mandatory is purely policy and protocol driven. The system will not reflect anything differently in the terms of mandatory versus option, so there will be no system mandates that you have to enter partnership data if a program mandate is put in place, this is something that is encouraged through relationships between fundees and their respective department or program.
The system will not change either way, so the system partnership fields are available for use but there will not be a mandate or a mandatory flag saying that these have to be entered or saying that these were missed. They will have to be managed by organisations and their grant managers and the program that they are delivering.
There are some programs where the partnership approach will be mandatory such as Try, Test and Learn and Reconnect and it really does, enhance and show the commitment that the department has towards the shift in focus from outputs to outcomes. As James said those fields, they won’t be actually mandatory when you’re building the system there, but again, more information will be on the Data Exchange.
‘How does an IT vendor get access to the reference data?’ That’s a great question and there are a couple of options. The reference data is available to download from the Data Exchange staging environment which is actually aimed at IT vendors, so we encourage all IT vendors to get access to this and they can download the reference data from exactly the same spot as the live Data Exchange. Alternatively, IT vendors can ask to have this file sent to them by the organisations and we should point out there are two different levels of reference data. There is the actual DEX reference data which lists the static fields in DEX so that’s the fields that have the same set of values for all DEX organisations, your country of birth codes, gender codes, stuff like that, that applies to everyone.
The other file is a little bit more dynamic and that is outlet and activity data file so that reflects the required codes for a particular organisation and no one else. IT vendors can download examples of this from the DEX staging environment but they will only reflect that test organisation. For live DEX this information will need to be provided by the organisations to the IT vendor.
Terrific, thanks James for that and for responding to all those questions. Again if you do have any further questions after the webinar has finished, then please contact our Helpdesk. We are going to leave the webinar open for another 10 minutes for any other residual questions that come through. So we will actually switch off our audio so there won’t be any of us speaking, but if you type your questions through you will receive a typed response, so please send those through.
For any of you that would like to leave the webinar now you can do so and we would like to thank you for your attendance and participation. This will conclude our session today and this webinar has been recorded and we will post it on our Data Exchange website under the Webinar library tab. Thank you. Enjoy the rest of your day.
This webinar provides IT specialists an opportunity to find out more about the IT requirements for the upcoming Data Exchange system enhancements. The webinar also provides information to assist IT specialists with submitting data using the system-to-system transfer and bulk upload methods.