Try, Test and Learn Client survey

Transcript for Try, Test and Learn Client Survey webinar

Slide 1

Welcome to this information session for the Try, Test and Learn, or TTL client survey.

My name’s Adina, I work in the Data Exchange, and it’s my pleasure to be taking you through this webinar. I’m joined by my colleagues to help answer your questions through the session.

For this session, we are meeting on the many traditional lands around the country and I want to acknowledge traditional owners of those lands and pay our respect to elders past and present.

We’ll record and publish the webinar on the Data Exchange website. Please check back in a few weeks if you’d like to revisit or share this information.

Slide 2

Before we get started, I’d like to give you some tips on how to use the webinar functionality.

Throughout the webinar there’s a control panel that should appear on your screen.

Clicking on any of the grey ribbons will expand that content for you.

Also, please remember to check your audio – your microphone symbol will be red, this indicates that your audio is on mute. If it is not we’ll all hear what is happening on the other side of your microphone.

Slide 3

When the control panel hasn’t been used for a period of time, it automatically minimises from view. To expand the panel again, simply click on the orange arrow and the panel will redisplay on your screen. On screen, you can see the question box, which you can type into.

Questions are a private function, only you can see your question and the answer from the team here.

You can also see the handouts ribbon, I have three handouts for you today. You can also see this information in your starter pack sent by your funding arrangement manager.

Slide 4

During the webinar we’ll talk about the purpose of the TTL client survey, your organisation’s role in survey delivery, and show you how to navigate the system in a walk through demonstration.

Let’s move to the first agenda topic, an overview of the TTL client survey.

Slide 6

The Try Test and Learn Fund, known as TTL Fund was announced in the 2016-17 Federal Budget as an initial response to the Australian Priority Investment Approach to Welfare. It develops and funds new or innovative policy responses aimed at improving workforce participation, or capacity, for groups at risk of long term welfare dependence.

TTL is designed to gain insight into what works to improve the lives of clients from priority groups.

The overarching evaluation of the TTL Fund ($96.1 m) and the evaluation of the tranche 1 programs is conducted by the University of Queensland Institute for Social Science Research. The evaluation of the tranche 2 programs are expected to commence as programs are gradually mobilised.

The Data Exchange is used by both internal DSS staff and the external evaluators to monitor the implementation and progress of TTL programs. TTL funded organisations are required to enter the data on a regular basis to enable issues to be identified early.

As many of you would be aware, de-identified data from the Data Exchange can be linked with the Department of Human Services (DHS) administrative data. This helps to identify basic welfare receipt information of those individuals that are receiving a service. To ensure the client’s privacy is protected, this linkage is done using a Statistical Linkage Key. This de-identified linkage work is particularly useful for the TTL fund because it specifically targets clients at risk of welfare dependency. Linking to the DHS welfare payments data enables the Department to look at data that is not currently collected through the Data Exchange, such as the individual’s history of benefit receipt, aspects of housing instability, caring responsibilities, reported earnings, and receipt of crisis payments.

I would like to emphasise again that this linkage work is being done on a de-identified basis, it is in line with the Data Exchange privacy and consent provisions. If you would like more information about how the Data Exchange protects client privacy, I encourage you to visit the Data Exchange website and read our Privacy Impact Statement.

Slide 7

The TTL client survey’s purpose is to support the evaluation of TTL fund programs.

The survey includes

basic information about each person,
their current involvement with work or study,
ideas about how they see themselves (such as goals and access to support), and
things that make it hard for them to be involved in work or study (including physical and mental health issues, housing and care responsibilities).

The client survey data is intended to complement information being collected through Activity Work Plans, SCORE outcomes and priority requirements data, being reported via the Data Exchange and other evaluation activities. This survey is each client’s opportunity to give the department their views on the service they have received.

Slide 8

This is an online survey that can be completed on a desktop computer or mobile device on your premises. Alternatively, where the client has access to a mobile phone or their own email address, they can use these tools to complete the survey in their own time.

The online survey includes ‘text-to-speech functionality’ which can be used to help clients with vision impairment or low literacy levels to complete the survey.

We ask that clients be invited to participate at the beginning and end of their time in the TTL program. An optional third occasion, if you choose, would be in the middle of their time in the TTL program. The multiple invitations will help where clients are at, as they move through the program and not lose response if they drop out early. It provides information about how clients are changing during their access to the service.

Slide 9

I have a Poll for you today. This is just to help reinforce your learning during the session. I’ll ask four questions as we move through the material. I’ll give you the correct answers at the end of the webinar.
The first question is:

Q1. How can clients access the survey?

On a laptop only
Paper copy only
On a smartphone only
On a Laptop, PC, Tablet or Smartphone.

Just thinking about the material you have heard and selecting the correct response.

Slide 10

Organisations are asked to invite all eligible clients of TTL funded program to take part in the survey. This includes all clients over 18 years old, or mature minors, accessing a TTL service.

A mature minor is an individual who is under 18 years old but has a level of maturity that means they are able to make an informed decision in the context of this survey. That is, they are able to understand what the survey is asking of them and provide informed consent. This includes the ability to understand the types of questions that will be asked as well as the voluntary nature of the survey.

If the client is less than 18 years old and is not considered a mature minor, they should not be offered the survey.

It’s important to note that the survey is voluntary and clients can skip any questions they find difficult.

Slide 11

There are some times when it’s not appropriate to offer the survey to the client, particularly where the client is not able to offer informed consent. ‘Informed consent’ means that they have the understanding that allows them to know:

the types of questions that will be asked;
that the survey is voluntary;
they can exit at any time; and
their response to the survey will not affect the service they receive in any way.

Organisations are asked to use their professional judgement when assessing whether a client is capable of giving informed consent. If you do not believe the client is able to consent, then you should not offer the survey to them.

In cases where the client is in acute distress, the client’s needs must be addressed first. This survey is not offered at that session and may be offered at a more suitable time, when the distress is resolved.

The survey is only available in English, meaning TTL clients need to have a level of English that allows them to be able to understand and respond to questions in the survey. This can be fluent in either written or spoken English.

Slide 12

This brings us to the second question from our poll ‘What is the eligibility criteria for clients to participate?’

There is no eligibility criteria.
The client must by over 18 years old accessing a TTL funded program.
Clients that are over 18 years old, or are under 18 years old but considered a mature minor.

Just selecting the response, which you believe is right.

Slide 13

What is the role of an organisation administering the survey? Please note, your Funding Arrangement Manager will be providing you with a Resource Kit explaining all of these steps in detail, but the webinar is intended to give you an overview of that material.

Slide 14

Your role is crucial in delivering the TTL client survey.

We ask that you:

Engage clients in the survey drawing on your knowledge and experience to modify the messages for introducing the survey to your clients.

Individual clients that can participate in the survey and the best point in their service journey to offer the survey to them.

We also ask that you understand the barriers to participation and help identify ways to overcome these barriers, noting some clients may need support during and after the survey.

At a practical level, you will need to facilitate participation by providing an appropriate space, and device or invitation for clients to complete the survey. You’ll initiate participation through a login page. The client will need to use this space and internet connected device to undertake the survey alone, unless they seek your help.

We’ll now go through the survey administration in more detail.

Slide 15

Your organisation will manage the clients who are to be invited to complete a survey. We ask that the invitation occurs at the beginning and end of their time in the TTL program. With an optional third occasion in the middle. The multiple invitations capture the changes clients report during their access to the service.

The first survey invitation should be offered at the beginning of the service or as soon as possible after intake. Generally, this should be within the first month of their time with your service.

The second survey invitation should be offered when the client is roughly in the middle of their time in the program or where a start or end survey will not be possible. This is a great time to pick up those clients that were unable to participate at the start of service delivery, due to their start date in the service prior to survey or distress at the time. Current clients would be offered the survey now, at the mid-point, as the survey was not available when they started.

The final survey invitation should be offered as close as possible to the end of a client’s time in a service.

Please note that the questions remain the same in each survey, a client completing the survey three times will be answering the same questions three times, to get a sense of what has changed for them as a result of service delivery.

Slide 16

This slide shows the steps for administering the client survey. We’re now going to walk through each step in detail.

Consider where the survey will be delivered
Invite the client to participate
Offer the ‘Participant Information Sheet’ to clients.
Seek the client’s consent to participate
Log in to the survey
Support the client (if required)

Slide 17

As mentioned earlier, TTL organisations are asked to administer the client survey as part of service delivery. It’s recommended that the survey be administered on your premises or wherever service delivery occurs, and that you tell clients about the survey as part of the intake/booking process. If appropriate, you may wish to ask clients to come a little earlier to their appointment, to allow time to complete the survey.

Step 1. Set up a private location where clients can complete the survey.

It is recommended that you set up a private area within your office for clients to complete the survey. Ideally this would be a separate room, however we recognise this may not always be practical.

Where the client prefers, they can complete the survey on their own device, and at a location of their choosing. Clients can choose to receive the survey through a Quick Response or a QR code or email link, which your organisation can generate for them. We’ll run through this process shortly.

Please note the survey is online, it can’t be delivered via paper. This is because the online survey system automatically generates a client’s Statistical Linkage Key. This ensures that survey responses are de-identified and protects a client’s privacy. This can only be done through the online survey tool.

Slide 18

Step 2. Research shows that the way practitioners introduce the survey to clients has a large impact on whether clients participate.

The invitation can be a verbal discussion between the practitioner and the client.

The Department has prepared a script that organisations can use to offer the survey. This will be sent in the Organisation Resource Kit, as part of your starter pack.

While we ask you to invite all eligible clients, participation is always voluntary, and the client can change their mind and opt in or out at any time.

Slide 19

Step 3.We recommend your organisation has copies of the ‘Participant Information Sheet’ printed for your clients to review. The Department is unable to provide printed copies for you.

It’s important that you offer this information in your invitation to clients and when you seek their agreement to receive an invitation.

Slide 20

We’re now going to take another quick poll. Just a reminder that I’ll give you the answers towards the end of the webinar. Question three is:

Q3. Do clients have to participate in the survey?

Participation in the TTL client survey is completely voluntary, and clients may withdraw at any time.

Clients must always participate in the TTL client survey.

Consent will be obtained from survey respondents by an implied consent process.

When inviting the client to participate in the survey, it is critical that you outline the information in the ‘Participant Information Sheet, to help them make an informed decision and verbally agree to receive the invite.

Just thinking about the material you have heard and picking up the correct response.

Slide 21

Step 4. Consent will be obtained from survey respondents by an implied consent process.

Slide 22

Step 5. To deliver the survey, you’ll need to log in to the client survey website and enter your organisation’s details. The log in details, including the website address and password are in the TTL starter pack, sent by your Funding Arrangement Manager.

Logging in is necessary to link clients to the particular TTL organisation and outlet they visited.

You’ll then enter your organisation, outlet and program activity information, and your password. You will need to select the relevant Try Test and Learn (Tranche 1 or 2) activity from the drop down box.

This log in screen is very brief and simple for your staff to follow.

Completion of this step launches the survey for clients. You can also use this login page to generate an email link or QR code, if the client prefers.

Slide 23

Step 6. The safety and wellbeing of the client is always paramount.

If clients are completing the survey on your premises, please make sure they are able to access supportive people or a support service in the event that the survey raises concerns or is distressing for them.

A copy of the ‘Participant Information Sheet’ must be emailed or included in their invite. Please ensure they can also find support if they need.

Slide 24

Our last poll question:
When will clients be asked to complete the survey?

Once only during their participation in the program.
Twice at random points in the program
Twice, with a possible third invite. At the beginning and end of the service, and in some cases in the middle of the program.
Never

Just pick the response that you believe is correct.

Now, I would like to go through the answer with you now before we move on. The first question was How can clients access the survey? The answer here is:

The clients can access the survey in Laptop, PC, Tablet or Smartphone.

Question two asked was ‘What is the eligibility criteria for clients to participate?’

The answer there is the Clients that are over 18 years old, or are under 18 years old are considered a mature minor. And we ask you to use your professional judgment when accessing the client and deciding whether to invite them. But we do encourage to invite all clients to receive an invite if they are eligible.

The third question was asking ‘if clients have to participate in the survey’?' And the answer there is that Participation in the TTL client survey is completely voluntary, and clients may withdraw at any time. But again, we do hope for high participation or as near as 100 percent as possible. We ask that you invite all clients and it is up to the clients that they respond or not.

And the last poll question asked ‘When will clients be asked to complete the survey?’

And I have mentioned a few times now, that is: Twice, with a possible third invite during their time accessing the program. At the beginning and end of the service, and in some cases in the middle of the program.

Slide 26

All of this valuable data you are collecting will be shared back with you in the form of an aggregated de-identified report. This report will help identify relationships between service delivery and your clients’ progress in work and study.

I would like you to note, you won’t be able to access individual client responses. This is to protect client privacy and reassure clients that their survey responses will not impact their access to service delivery.

The report is currently undergoing development. The release will be dependent on having a big enough sample size of client responses, so we will need to build completed responses before we could offer you any information back.

Slide 27

I will just take a moment for the set up though the walk through.

You will notice that this survey page has its own website and that link will be sent in your starter pack through your funding arrangement manager.

I’d like to point out that the survey will run on Apple, Android and Microsoft platforms.

Today we’ll be looking at the training database, so all the entries are fictional. Just to give you an idea of the fields as we move through the survey. I’ll go ahead and login to the training database for this walkthrough.

You can see we selected organisation, outlet name, program activity and we will just login with a password here.

Your login details are sent by you Funding Arrangement Manager, and can be used by anyone in the organisation to set up the survey for the client.

I have mentioned earlier that in this page you can generate a Quick Response or QR code or email link. These are really useful depending on what the client would like to use, to complete the survey. I think of most interest here is the email link and you can see if you select it, you get this option to generate that link. You don’t need to do this for every client, you can generate that email link and it will be saved to your clip board and then you can paste it to your invitation email and you can bcc (Blind Carbon Copy) clients you would like to send this out to. But for training purposes I would like to take you through the web survey. I am just going ahead and login here.

This is the Welcome page, this is just the organisation seeing it, your client won’t see this page.

This webpage has guidance for you to give it to your clients and there is a range of information you can look at. I would recommend you to print some of this, particularly the participant information sheet for your clients to have when they start the survey.

I would just like to point out the ‘feedback report’ here. This is about metadata, this is not about client responses. I have opened this up and you can see this is for my training organisation, there is no real data in here and I have only completed one training survey.

Hence, really it is not meaningful, until you get to add a number of completed surveys your clients have been through and submitted their data, that is where you will find the most value for this report. This gives an overall summary of how many completed, generally how long it took clients to go through the survey. You can see as I was doing some training it took me quite a bit longer than I would expect clients to take. It talks about the times that might be useful for you to look at whether clients are doing it late at night or whether they are doing it in the day. It will help you to tailor your invitation. Now here we can see the type of devices that are being used to complete the survey. At the moment there is not a lot of information but as your clients go through and submit their responses, this will become very valuable in understanding how your invitation is working and how your clients behaving but you won’t see any identifiable information or survey responses here. I will go ahead and close out.

Client survey page: You can see here ‘Begin Survey’ button. When you select this button, you will be rolled to the first page the client will see. This is the point where you will hand the device to the client and they will go through and start responding to the survey questions. There is a range of information for client just to make that informed decision about whether to participate or not. Ok, I’ll go ahead and start this survey.

Your details: As this is a fictional client in our training database, I am just going to make up a name here, but of course, we will encourage your clients to use their real name and certainly use the name they are registered for your service with, so those responses can be linked.

We are in the actual survey now:

Survey questions: There’s a question at the start of the survey which asks the client if they received help in answering questions. This does not refer to technical support, this is about who helped them understand the question and formulate a response. In my case, I am doing this survey myself.

You can see we have our first mandatory question. This is indicated by the red asterisk. Clients cannot move on until they answer this question. For most of your clients they will be near the middle of the program, as the survey wasn’t available during their first month of service, hence we are offering at the optional mid-point.

In the next page we have a range of questions here about the program and whether the clients agree to those statement.

These next questions are not mandatory, I am going to skip right through as this is a training. We do strongly encourage the client to answer all questions though.

Now here we have another mandatory question. We are asking where the client lives and it just takes a second to bring up the suburbs for them. The clients can select the most accurate answer there. We have some questions about language and heritage. We will just move on through.

This has brought us to the first of the progress pie chart, the idea behind this is encouraging clients to go through to completion. We did a little bit of research on it and we found that this is the best method for encouraging clients to continue. It is just a very simple visual indication on how far has the client moved in the survey.

Another useful function is the play button, this converts text to audio. Some clients may find this useful as some clients can be fluent in spoken English but not so much in written English, so this helps them as they move through.

You can see we are continuing to make progress through the survey, and it’s not that time consuming at all. And this pie chart is just encouraging us to continue with our progress.

Some questions about how the clients sees themselves, again these questions are not mandatory. We can just skip through, of course we would encourage clients to complete as many of these questions as accurately as possible, as they can complete the survey. But, for demonstration purpose I am skipping through.

I’d like to point out there is this option to exit the survey. If the client clicks on it, it will just bring up another screen, which will just bring up the type of metadata that is recorded and you saw it earlier when we looked at how long it took the client to complete the survey. Back in the office here, we can see what questions they were exiting out of most often. This helps us to improve the service which we offer and understand how clients are using this tool. They can of course just exit out of the browser and that will close the survey for them. If they do exit before completion, they are unable to pick up where they left off. They will need to start the survey again.

This has brought up another progress pie chart, the visual representation of how far we have moved through the survey. We are continuing to move progress here and working through the survey. A range of more questions that I am going to skip through. You can see clients are just asked to select options whether they strongly agree, agree, neither agree nor disagree or strongly disagree. They are fairly straight forward options for the clients.

They can move back if they like and revisit any of their answers. However, if they do exit the survey it is not saved, the data is not submitted and they will need to start over again.

I will keep on moving through. We are close to finishing the survey now. And this is a nice way to end the survey, we have these final few positive questions for the clients and they can go through and respond to those.

The survey is now completed and we can select submit. This puts our data to the database. If we don’t do these step the survey responses will be lost and it will not be submitted in the database. It is important that they do that.

Once completed and submitted, they just receive this message thanking them for completing the survey, that all of their data is securely stored and DE identified. And here we have the same message which appears, if they select the exit button. Just giving them some information should they be distressed or upset by any questions.

Slide 28

That completes our survey demonstration. As mentioned before you will receive your login details from your funding arrangement manager shortly. You will be able to go in and start offering the survey to your clients.

I’d like to go reiterate the key points of the webinar today, these are:

Offer the survey to as many clients as possible, keeping in mind the inclusion and exclusion criteria.

We ask you to use the Organisation Resource Kit to help you deliver the survey in an ethics approved way. Really, this is as simple as following the advice set out in this webinar, and not offering paper surveys or inviting clients in acute distress. It is always best to seek agreement from the department before you depart from the set approach.

Be mindful of ethics! If the survey is offered in a different way, you may be jeopardising this valuable data and its contribution to the TTL evaluation.

Slide 29

Where do you go if you need help in delivering the TTL client survey?

The first point of contact for help is:

Your funding arrangement manager – they will be able to provide guidance on data entry, grant requirements and program specific information for the services you deliver.

TTL evaluation – this team can help you with any aspects of the TTL evaluation. Please note we have the email address for assistance.

You may also wish to speak to data exchange helpdesk as they will be able to help with Data Exchange technical issues.

In addition, I would like to highlight there is also the Data Exchange website – you’ll find training and policy support material on the website. We strongly encourage you to subscribe to receive the latest Data Exchange news and updates.

Slide 30

On behalf of the Data Exchange, I’d like to thank you for your participation. If you’d like further information on the survey, please review the documents sent to you in the starter pack, by your Funding Arrangement Manager.

Alternatively, we’ll stay on a little longer to answer any more questions you have, keep sending them through. Once again thank you for your participation. Goodbye

How can we help you?

Training Resources

Technical Support

Data Exchange helpdesk