Talk To Me Like One of Your LLMs

I am captivated by the rise of AI and the immense potential it offers to reshape the many computer platforms used in creative + entertainment technologies. Large language models (LLMs) are incredible tools, and I employ many in my own work. The ability to rapidly brainstorm ideas and open up new areas of innovation is undeniably powerful.

But with these advancements, I also recognize that the landscape of digital labor is undergoing fundamental shifts. Like any technological revolution, there will be a simultaneous mix of the resistance to change, and an exaggeration of the change that is coming. I don’t fear AI or reject its use as a creative tool. Nor do I expect it will displace human powered creativity. However, I’m concerned how we as a community adapt to the use of AI models, how we will continue to innovate in our craft, and how we ensure that we are leaders in this new era, rather than passively following this disruption.

I’m also acutely aware that any labor discussion about the entertainment production industry and AI is intertwined with the very bumpy recovery from the pandemic. There is an undeniable depression in current employment opportunities. Did AI squeeze you out of a job or is it just garden variety capitalism? And how can we tell? We are a poorly organized community of professionals with a wide range of skills across numerous fields of work. How do we promote our own AI agenda when our labor is already undervalued? Whatever you feel about AI or its impact on your work, we need to be navigating these changes together, thoughtfully and with foresight.

With that in mind, I want to unpack a thread of future AI discussion points by combining these labor questions with my thoughts on the complexity of design communication. I can cover these topics quite nicely with a healthy rant about prompt engineering.  

What was your reaction to learning about prompt engineering? Inspired? Enlightened? Insulted? Confounded? My reaction after reading through a prompt engineering strategy was, basically, “You have got to be fucking kidding me.” Don’t misunderstand me; I’m not frustrated by the desire to use AI. I’m frustrated by how much thought has been put into creating effective prompt strategies over the last two years, when I very rarely have been given such clear instruction in a 30 year career. Of course LLMs are fucking successful; we’re drawing them a damn map of what we want. 

The frustration I feel is not subtle, but it is very nuanced. I haven’t been supplanted by a superior work product. I wasn’t replaced for a service I once provided. It’s not like an LLM was celebrated for an idea that was originally mine, though I do recognize the clearly problematic derivations of other’s work that have occurred. Put a pin in that particular thought while I draw attention to the problem I see: it’s us. Human fascination with replicating ourselves in a machine – that is the problem. We’ve so easily rushed to adopt AI models into our work and into all of our software. We’ve followed guidance on the best way to communicate our tasks to tools. I think it’s important we pause to ask why we do this for the machines only and not each other?

Can we take a moment and consider how much human output would be improved if we applied prompt engineering to creative communication? If we thoughtfully outlined a necessary task, the character of that task, the steps we want taken in the completion of that task, and defined the parameters for success for every other human we interacted with – wouldn’t human output improve as well?

Think about it like this: prompt engineering is the process of designing inputs for artificial intelligence models to produce desired outputs. It involves crafting a set of instructions that help AI models understand the intent behind the query and respond in a meaningful way. Or at least that’s what Google’s AI Overview says it is. From my point of view, prompt engineering is simply outlining a task for completion. That task can be anything from writing a poem, to analyzing a data set, to summarizing this essay (I’d do it, so I assume you are too). The more detailed the prompt, the better your result. There are a number of specialists, service companies and schools of thought on how to design prompts for LLMs. Why has this discipline exploded for software, but not wetware?

Clearly, I’m not going to argue that prompt engineering is bad or wrong. Instead, I want to understand the social factors that enable clear communication when using AI for creative development. I would like to recognize our own challenges in communicating creative goals to each other in the hope we collectively come out of this AI growth phase as better humans. Yes I can hear that sounds a bit daft, but you’ve gotten this far in the article so you might as well stick with me. 

I’ve spent decades in the collaborative creation of entertainment projects. I work in the technology that supports theater, film, television, live music and creative arts. My professional success is dependent on clear communication and filing in gaps where a lack of information exists. However in this field, my colleagues and I daily experience the meme of poor design & technology communication. 

@fentifriedchicken

Basically every day in Corporate America IB: @Cas3yart #Working #Finance #Consulting

♬ original sound – Joe Fenti

Have you ever felt this way? You do your best to respond to the limited information in the request as made, only to be disparaged for your lack of absolute understanding. Perhaps you possess an entire approach to your work that is designed to serve clients that are encouraged again and again to fail at communicating with us. Maybe you even thrive in the lack of feedback.

Some of our clients believe they won’t need most of us anymore if the promise of their AI future is to be believed. And they wouldn’t be entirely wrong. Our jobs are going to change, but luckily for us in the creative technologies: 

1) good prompt engineering is hard

2) our clients don’t always know what they want

3) the AI future being sold to our clients comes from entities who are likely to need our services in the not too distant future. 

No one is replacing our entire discipline with AI models anytime soon. But there is going to be massive change we need to be prepared for. We can prepare by learning more about prompt engineering and applying those strategies not only with AI, but with each other.

As a creative technology producer, my primary work function is to hear what the project goals are, reason a path to meeting those goals, and communicate that information to others in a way that is optimized for their understanding. Whenever I think I’m quite accomplished at this task, I am humbled by the ease of miscommunicating the most basic of goals. It is a personal mission of mine to continually refine the way I communicate, and the way I collaborate on creative endeavors. Prompt engineering immediately resonated with me as it solves so many communication issues that occur in collaborative projects. 

Look, interacting with humans is messy. We are bad listeners caught in our own bubble of insecurity wrapped in an ego driven quest for professional affirmation. Asking someone in a creative and/or technical field to “make it more blue” is a portal to childhood trauma. There is a clear advantage to collaborating with a tireless results machine that has zero personal hang ups. And it shows.

When it comes to design development, I would argue that the output from LLMs suffer from being un-messy. Collaboration is interesting because it is human. Our humanness introduces mutations in creativity through misunderstanding, sometimes to absolutely glorious results. We can see and hear the difference between human made and machine made. That might not always be the case. 

Ultimately, I see the potential for the use of prompt engineering strategies for more than just AI models. I see them as an approach to creative communication. What works for an LLM, can work for any collaboration.  I also see prompt engineering strategies as a pathway to supercharge mentorship for junior team members. These procedures provide clear guidance to those new to entertainment production, as well as reduce the barrier to entry on the latest entertainment technologies. 

——————–

Let’s go back to the pinned point from earlier: derivative work from uncredited source material. How are we in the creative technology fields going to shape a future that includes our labor, ethical use of our work product, and well executed use of AI? I take the point of view that AI is unstoppable. The labor disruption is already here and we all need to be considering how we adapt. The international race for model training data has taken priority over the model companies’ ability to properly credit/pay/protect individuals for the misuse of their works. Any legal fights attempting to correct this oversight/error/theft will drag out for a longer amount of time than some of the current predictions for AGI. This race is unstoppable.

Instead, I want to use this platform to ensure that creative technology professionals are seen as leaders in the discussion of AI use and ethics. Our community will be at the forefront of AI tool use for all manner of entertainment production, from design to content generation, image processing and show control.  For all my comedic discourse on the frustrations of prompt engineering, the best prompt engineers are those that understand the strategy and working environment. Experience is the best prompt expertise. That experience should be used to work with models and to train future experts. We are those professionals. The future is us.

——————–

This article is a starting point for a series of articles on AI in creative technology. I welcome collaboration or others to write about their experience with AI. There is a platform at frame:work and at Substack to share and discuss these ideas. I hope you will join me. Here are some topics I would like to discuss further:

The Labor Squeeze

While AI allows productivity per individual to explode, job opportunities are in decline. How do we enjoy our newfound AI-juiced capabilities and balance the employment issues?

Hybrid Work Models

How do we mature junior professionals into leadership roles if junior level tasks are being sourced to AI? Let’s build a working standard for hybrid teams to protect industry knowledge for future practitioners. 

Mentorship and Apprenticeship Programs

How do we fund opportunities for new professionals to get practical training? Can we engage the enablers and corporate users of AI to invest in mentorship?

Collaboration Training

What are the social pressures that complicate collaboration? How do we adapt prompt engineering to human communication when working in collaborative projects? 

Continuous Learning and Upskilling

How do we create opportunities for experienced professionals to learn AI tools and build their own prompt library?

Preserving Craftsmanship

How do we document current workflows and maintain support for a craft approach to creative technology that new professionals can learn from the ground up?

Ethical AI Deployment

How do we engage clients and ourselves to adopt working standards for ethical AI use and hybrid work models?

Transparency and Accountability

How do we set a working standard for documenting AI use in a project, or what resources and AI used to create work? Where is the line for derivative works without proper credit or compensation to original creators?

Bias and Fairness

How do we set standards to address issues of bias in AI models, which can perpetuate stereotypes or exclude certain groups?

As an addendum, I will say I’m inspired to lean hard into using AI by the software development community. I cannot think of a group of people working as hard to build tools that are likely to replace their labor as quickly as an AI software engineer. If that professional community can approach the future fearlessly, they must see an outcome worth the risk. I intend to join them.