Noah D. Arney

Tag: technology

  • This is how you get an academic survelence state.

    Sometimes great ideas start throwing up red flags very quickly.

    I love data. Data is my jam. It’s why I’m currently on the Strategic Enrolment Management group at my university and why I’ve been part of a similar group at two previous institutions. Data and assessment are a great way to enable us to make things better for students. So I’m coming at this as the type of person that the author of this piece want to recruit toward their way of thinking and I need to tell you OH HELL NO.

    Higher Education Needs Its Own Version of Moneyball

    Lets start with part 1 of the premise:

    Higher education needs its own version of moneyball—a set of active, predictive and creative measures that can be deployed to improve student outcomes and fulfill their promise of student success.

    Makes a lot of sense, and I fully agree. This is what a culture of assessment and SEM looks like. It’s amazing and I am here for it.

    And then in the next sentence the red flag gets waved high:

    Postsecondary institutions must be able to collect and instantaneously analyze student progress data and have intentional plans for adjusting in the moment to the needs of their learners.

    There’s the oh hell no moment. Moneyball style analysis uses publicly accessible and consensually given information and visible information to work. This asks us to tie every moment of a student into a machine.

    Ok, lets dig into the points that support the premise.

    1. part time learners don’t complete their programs as often so they should be assessed more
      • But part time learners often have other things outside of school like work or family responsibilities, which normally means that they are more likely to be financially disadvantaged students. So you’re asking us to have more surveillance of those who we know are already over surveilled and over policed?
      • Also there are often reasons why students pick their course load. Supporting them in increasing that course load is a great idea, forcing them to increase their course load without supporting the reasons they didn’t think they could is awful.
    2. Productive credit hours as a measure – do students take more classes at certain times or days, or are there too many gateway classes preventing moving forward
      • This one makes sense and I’m here for it, but this is just proper scheduling and doesn’t require real-time analysis, just semester based analysis which is what SEM already does.
    3. Predictive metrics
      • There isn’t any information given for this one so I’ll have to make assumptions. “This planning starts with insights that enable institutions to identify opportunities for accelerating student progress and predict the efficacy of those interventions on retention and graduation rates.” Predictive metrics mean one of two things:
        • Constant surveillance of students (how often do they attend events, library use, computer on campus use, assignment submission times, in-semester grades) which is problematic and sounds like a surveillance state.
        • Assumptions of students based on statistical models which often break down when applied to the individual. For example, in the USA, ~1-5% of adults are diagnosed with ADHD, but ~25% of adults in prison in are diagnosed with ADHD. Does that mean ADHD is a predictor of crime, or that people who are institutionalized are more likely to receive a diagnosis? What about when you find out that ~20% of adults in post-secondary education have ADHD? Are the predictions based on data always reliable? They might be in aggregate, but the idea here is to take that aggregate and apply it to the individual. That’s like looking at a normal curve and saying “well, people over 7’4″ don’t exist” when we know for a fact that they do, but they are rare.
    4. Make the stats open to all
      • Again, statistics in aggregate about the student body, even when looking at relatively small groups, is a great idea. How many students take classed in X department, how many of them pass it, how many are International student, etc. This data is important for SEM to identify gateway classes, problem pathways, programs that are missing something, or departments that are under-enrolled. But using it in specific is dangerous.
      • Do we really want a professor to be able to know how long a student takes to complete an assignment in someone else’s class? What about whether or not they use the library? How much data do you need about the specifics of someone? I don’t even like that my LMS lets me know how long ago a student accessed the system.

    Where this article is right: SEM is the way forward. Data is important, and needs to be viewed by as many people as possible. Universities and Colleges are filled with brilliant people, getting more eyes on a problem with the relevant data means more potential solutions.

    Where this article is scary: implying that we need to feed all data about students in real time into a data analytics system, and then turning that into and using it as predictive metrics of success.

    It feels like an article written by someone who isn’t seeing students as people, but as bundles of data that they can access. That way lies teaching machines, but the way forward toward better developmental and lifelong learning outcomes for students (regardless of academic outcomes) is through relational connections on the individual level that are supported by using data on the macro level.

  • AI Hasn’t Cracked Persuasion Yet

    I spent some time today playing with ChatGPT between meetings. I figured that I should see what it could come up with when asked about job application documents. It’s idea of what a resume looks like is rooted in the resumes of yesteryear, but it’s not so bad that I’d throw it out, just guide the writer in how to explain their impact in the prior roles.

    However, what I really wanted to see was what it could do for a cover letter. I started out simple by seeing if it knew the formatting, and the result was… well I guess that counts?

    Read more: AI Hasn’t Cracked Persuasion Yet

    Dear [Hiring Manager],

    I am writing to apply for the [position] at [Company]. With my extensive experience in [field], I am confident that I can quickly become an integral part of your team.

    In my current role as [position title], I have been responsible for [responsibility 1], [responsibility 2], and [responsibility 3]. These experiences have given me an understanding of [industry] and a valuable set of skills. I also have experience with [software/platform], which could be beneficial to your team.

    I am very excited about the opportunity to join [Company] and believe I have the dedication, knowledge, and enthusiasm to excel in this position. I am a quick learner and would appreciate the chance to bring my skills to the team.

    Thank you for taking the time to consider my application. I look forward to hearing from you and discussing my qualifications in more detail.

    Sincerely,

    [Your Name]

    It kinda looks like a narrative version of a resume though and that’s not the point of the cover letter. The broad strokes of the formatting are correct but it’s as if the system doesn’t know the point of the format. As you’ll see, that was the theme of my tests.

    Next up I gave it a job posting (I’m redacting certain info because it was an internal posting)

    Dear Hiring Manager,

    I am writing to apply for the Financial Analyst Co-op position at XYZ. With my educational background in finance, my enthusiasm for working in the capital markets and my strong analytical skills, I am confident that I am a perfect fit for this role.

    I have obtained a Bachelor of Science degree in finance, and I am currently pursuing a Master of Science degree in finance. My studies have provided me with the knowledge and experience to understand the complexities of the capital markets. I have also developed strong analytical skills, including the ability to analyze datasets, develop insightful commentary, and create analytical tools and initiatives.

    I have excellent verbal and written communication skills, and I can work independently and under pressure in a high-pressure, high-reward environment. I am also motivated to gain my CPA or CFA and am eager to take on increasingly greater accountabilities.

    XYZ is an organization that I truly admire and I am excited at the prospect of joining your team. I am confident that my enthusiasm and experience will be an asset to your organization. I look forward to discussing further how I can contribute to your team.

    Sincerely,

    [Your Name]

    It was able to pull some things in from the job posting, but it didn’t understand the required education and also wasn’t quite sure how things connected together. Instead what I got was a listing of skills in the job posting but with the words “I have excellent” ahead of it. It’s exactly what I expect to see of a generic cover letter from someone who doesn’t actually know what they have that’s valuable to the employer. There’s no real content in the cover letter it’s just reposting what the employer asked for. Also, Masters of Science in Finance? Really? That wasn’t asked for in any way, and the job is for someone with two years of education.

    After some tweaking I got a bit better of a cover letter

    Dear Hiring Manager,

    I am writing to express my strong interest in the Financial Analyst (Co-op), Capital Markets Performance Management position at XYZ. With my background in accounting/finance and reporting, my exposure to Capital Markets products and businesses, and my experience with analytical tools and managing large datasets, I believe I would be a great fit for this role.

    My background includes a B.Sc. in Accounting and Finance, in addition to experience in PowerPoint presentation, verbal and written communication, and strong analytical skills. I am a motivated individual who is comfortable working under pressure and can prioritize activities to ensure accuracy and timeliness of reporting.

    I am excited to join a team that is dedicated to providing trusted advice, helping clients thrive, and achieving success that is mutual. With the opportunity to do challenging work, take on greater accountabilities, and build close relationships with counterparts across the company, this position is a perfect fit for me.

    I am eager to learn more about this position and discuss how I can contribute to your team. I am available to discuss this opportunity further and can be reached at (000) 000-0000 or at [email protected]

    Thank you for your time and consideration.

    Sincerely,

    Your Name

    At least now it looks like a cover letter that I wouldn’t immediately throw out. But it still doesn’t understand that the point is to persuade the reader that they’re a good choice, not just restate what’s in the job posting with some platitudes.

    I’d share what happened with another job posting, but it was more of the same. Lots of platitudes and restating the job posting.

    I figured that it didn’t understand how to connect to the things in the posting because it didn’t have a resume, so I fed an example resume (we make fake resumes based off fictitious students to use as examples) and job posting in. The resume was one that would be just slightly indirectly connected to the job posting so that it was a bit harder. But it would still be something that I could easily walk a student through on how to make those connections. The AI on the other hand… well…

    Dear [Hiring Manager],

    As a 4th year Bachelor of Arts majoring in English and minoring in Psychology with an intention to pursue a career in the not-for-profit sector, I believe I am the perfect candidate for the position. My comprehensive knowledge of customer relations, developed through 5 years of high-caliber service in the food and beverage industry, combined with my strong team player and proficient interpersonal communication skills fostered by 3 years of experience as a community volunteer in event planning make me an ideal candidate.

    I am also highly organized and have excellent time management skills; maintaining a full course load with a cumulative 3.95 GPA while working a part-time job and volunteering. I am proficient in all MS Office Programs (Outlook, PowerPoint, Word, Excel). My experience volunteering at Anytown Shelter Thrift Store, assisting clients with community resources, as well as helping with social media posts and event management will be beneficial to the role.

    I have also presented a poster on sleep deprivation during student exams at the Psycology Conference in Anytown and have been awarded the Coop Student of the Year Award for my excellence and hard work as an Anytown University Coop Student. I am confident that my skills and experience make me an ideal candidate for this position.

    Thank you for your time and consideration and I look forward to hearing from you soon.

    Sincerely,

    Ella Pink

    My first thought was “YES, they got the idea” but then I kept reading. There were four connection points between the resume and cover letter, and the AI was able to figure out one of them. Even worse, it literally just dropped in wording directly from the resume without elaborating on it. And finally, there was something mentioned multiple times in different sections of the job posting (something a human would read as a flag that you need to mention it) but the AI didn’t bother.

    I ran that same resume through with a job posting for a psyc student co-op job with a child care company and a huge red flag went up for me. As soon as it had both a feminine name and childcare connected job the AI decided to stop overselling and be a little more humble about itself. It’s writing about how perfect it is for the job, over and over again, until it thinks it’s a woman applying for a job working with children then suddenly it’s talking about how it’s only a “good fit” for the role instead of “perfect fit” and now instead of talking about itself in an expansive way it talks about how it can provide “valuable support to the team”. Remember everyone, AI isn’t unbiased, it just lacks the ability to critically examine its bias.

    After all of that… well it’s a fun toy, and it’s better than nothing. If you had a resume with one of these cover letters and a resume without a cover letter I’d pick the one with the ChatGPT one. But it only understood structure and copying content, it didn’t understand what the job postings meant, or where the information in them was important or how to frame things in a way that was convincing.

    It seems like it may be a wile before AI has figured out what makes human communication convincing. But since that’s something a lot of humans haven’t figured out yet perhaps it’s not doing as bad as I think. Basically it’s the same issue that AI has with essay writing. The writing fits the structure it’s imitating, and mentions the things in the prompt, but it doesn’t have the ability to write in a way that makes connections between the material and the audience in a meaningful way, or to connect things between the material in a way that is convincing and not simply “hey these two phrases probably go beside each other”.

    I intend to use the output from this in my teaching on cover letters this year as it shows the difference between a good cover letter (which these aren’t) and a bad cover letter (which these aren’t) and a boring/formulaic/lazy cover letter (which these most assuredly are).

  • AI and Hiring

    A former twitter thread

    In response to this tweet:

    NEW: the insurance startup Lemonade claimed it was analyzing “non-verbal cues” like eye movements and speech patterns to reject insurance claims.

    then the company deleted a bunch of tweets, and now it’s saying “we def do not do phrenology”

    https://twitter.com/janusrose/status/1397602847064215554

    This is part of a trend of problems with companies who use AI to make decisions about people. Now, I’m not involved in AI ethics, for that you should follow Timnit Gebru, but my work in career development involves understanding how people assess other people.

    A problem in hiring is the role bias plays. If you ask 100 people if bias plays a role in their decisions about job candidates you’ll probably have 90 saying no. That’s actually one of the problems. Our brains do this wonderful thing where they say “I’ve seen that behaviour before and it meant they were lying”. But the problem is that the experiences your brain is assessing against are based in the culture you are within. The things it assesses are what people within your culture do when they’re lying. That may be different for those raised in other cultures. But your brain tells you that your experience is universal. And that’s not just about lying. It does the same for what means attentive, friendly, pleasant, combative, and dedicated. That means that when we’re interviewing someone for a job and we think about “fit” it’s very very easy to favour those raised in your own culture.

    So let’s talk AI. The cues that AI is told mean certain things are also culturally conditioned. Usually from how the system was trained. The problem is that AI can’t critically assess itself and say “wait, is that true or just what my upbringing says?”

    Now there are many employers who don’t critically assess their biases, and that’s a problem. But transferring those biases to AI and then claiming it’s unbiased because it’s AI is much much worse. So that’s where we are. AI remains subject to the garbage in garbage out problem, so pretending that it’s unbiased is untrue. What AI does is apply the same biases to everyone. That’s wildly different from being unbiased.

  • Automation and Career Development

    This was originally a twitter thread

    I’m seeing a lot of people talking about how people should go into HS only jobs or trades instead of university. Lets put aside that the unemployment rate for trades is often worse than jobs that require a university degree, instead I’ll tell you a story about the economy.

    I grew up in BC. And the alternative to university that was pushed when I was in high school was either the family farm (I lived in a farming community) or the lumber industry. FYI, this is a #CareerDevelopment story.

    It was the 90s and the lumber industry was strong. If you weren’t from a farming family the non-university jobs talked about were forestry/lumber, construction, plumber/electrician, and first responders. In the mill towns it was pretty much just forestry/lumber.

    The forestry and lumber industry was very people intensive. People to cut trees, people to plant trees, people to move logs, people to run the mills, people to support all of those industries, people to work in secondary industries (wood product manufacturing).

    So it’s the 90s and there’s about 100,000 jobs in the industry. They’re good jobs, well paid jobs. Most of them require no post-secondary or maybe a certificate.

    When I moved to Calgary five years ago the way people talked about the oil sands was exactly the way people talked about the lumber industry in BC when I was a kid.

    Now, I say that there were good jobs, and there were, but the number of jobs wasn’t really going up. And this doesn’t get noticed in the short term, but what it means is that the industry isn’t growing, which means the future won’t be bright for people trying to get in.

    Oh, productivity kept going up, the money the industry brought into the province kept going up, but employment was stagnant. That was never mentioned to teens looking to what their future could be though.

    So, what happened to that industry? Well, the 2000s happened. And at the end of it the industry had shrunk 50%. The 2000s were filled with talk about how we needed to “retrain” forestry and lumber workers.

    Magic bullet after magic bullet was proposed. The government started talking up trades, while ignoring the increasing trades unemployment rate. The jobs that had lower unemployment? Work that required a bachelors degree.

    FYI, here’s the Forestry & Lumber industry over 20 years. Yeah, it was bad.

    I talked in depth about the so-called Trades shortages about five years ago. TL:DR the only trades that have lower unemployment than bachelors degree requiring jobs are the ones that required two years of post-secondary apprenticeship program.

    That’s an important point a lot of people forget. Trades school in Canada is run through the same post-secondary system as Bachelors. The programs are generally 1/2 the length, but that’s it. So when I talk about post-secondary I mean Certs, Diplomas, Trades, and Degrees.

    What’s the point of this story?

    1. jobs that don’t require post-secondary are being automated
    2. once a resource extraction industry automates they never bring the jobs back
    3. people with post-secondary have an easier time changing industries when jobs disappear

    So, if you want to tell someone not to go to get a Bachelors degree, you’re still probably going to be telling them to go to post-secondary. That’s the way of the world now.

    As I look back on the people who talked up forestry when I was a kid I notice something. Most of them were let go when the mill automated or they changed industries in their late 40s. Some of them went to post-secondary then to retrain/reskill, and that’s a good thing.

    But here’s where it comes to Alberta. The same automation warning signs are there for the oil & gas industry. I had a student who I worked with a few years ago. He’d spent 15 years in the oil sands and decided to change jobs. Why? Because he saw the signs. He knew that his job was going to be automated in the next five years, so he decided to train now for the IT job that was going to replace 10 people who were doing what he was doing before.

    And that’s where we get back to #CareerDevelopment. Students need to learn not what the past industries were, but what industries are growing and flourishing. That is going to require post-secondary, of some kind.

  • Surface Phone?

    I love my Windows Phone. Unfortunately it’s time to upgrade and I can get a very good Android phone for $200 cheaper than the new Lumia 950. So I’m going to have to say goodbye to the Windows Phone for a few years. That being said Microsoft has done a good thing getting their ecosystem into Android, so although I’ll have an Android it will be running the same ecosystem apps I’m used to (oh Office, you make my life so much easier).

    But I’m not here to lament the cost of the new Lumia, but rather to look ahead at what might be. A Surface Phone.

    (more…)

  • I Am Not a Wallet

    I had this thought last night when trying out Sims Free Play on my phone. Everything takes forever to complete. While most of these pay to speed up games take about one to two minutes to do a task early in the game, just long enough to figure out what else you can do, Sims takes 10-120 minutes. So you set one task, close the game and come back to it later. It’s rather boring. But the game is fun when you’re playing it. Which is all designed to get you to pay for premium currency to make the game play faster. (more…)

  • Time for Inside Higher Ed to End Anonymous Comments

    There have been many cases against anonymous comments.

    Popular science removed their comments entirely for a good reason, as referenced in a New Yorker article:

    “The editors argued that Internet comments, particularly anonymous ones, undermine the integrity of science and lead to a culture of aggression and mockery that hinders substantive discourse.”

    Huffington post brought up To Kill a Mockingbird:

    “Lee’s basic claim is this: We are capable of doing far worse things to one another when we do not have to own up to the things we do.”

    The cases for anonymous comments tend to focus on the lack of effect restricting it will have as people are still willing to say terrible things in public under their own names.

    (more…)

  • On Technological Change

    We always overestimate the change that will occur in the next two years and underestimate the change that will occur in the next ten.
    -Bill Gates