REISSUE is a limited series on SeePlayLive where I transition over writing initially published elsewhere (e.g. Medium) that I think could be of interest to my Substack readers. Enjoy!

As someone with limited formal training in computer science or engineering or math, it can be daunting to try to make the leap into data science. It seems like almost every job description calls for someone with a degree in some hardcore field that, while interesting and important, never caught the fancy of those of us in the social sciences who prefer the study of people and societies over the study of cells, chemicals, energy, numbers, or what not.

It certainly felt that way to me when I was getting started in data science. After wading through countless job descriptions looking for someone with a degree in something I never studied, it was difficult to not wonder, Is there a place for people with a social science background in data science? or, I didn’t major in a physical science or engineering, can I really make it as a data scientist?

Well, having worked as a data scientist now, I’m happy to say there is and you can in response to the above questions! That said, it can be difficult to grasp at first how your social science background can be useful in data science, and there are definite some key things that are helpful to be familiar with that you may have never learned in school. I hope this article provides you with insight on how your social science background can help you be a strong data scientist and helps you determine what skills school never taught you you should pick up to smooth the transition.

Social Science Background

My own social science training comes through the field of communication, specifically the study of mass communication formats (broadcast media, social media, so on; specifically entertainment media in my case) and people’s interactions with and responses to them. But the label of social science can be applied to all manners of fields beyond my own — e.g. psychology, sociology, criminology, economics, political science, etc. In fact, graduates of such fields make up a big portion of bachelor’s graduates [1] each year. And among them, you are not the only one looking to break into data science, nor will you be the first to make it — so don’t despair!

Relevance of Social Science in Data Science

It actually shouldn’t be crazy to realize that social science provides an exceptional background to move into data science. At the core of any social science is investigating how various stimuli or elements of people’s (or groups’) background, thinking, or behavior are associated with said people’s (or groups’) other thoughts or behaviors or outcomes or characteristics, and social scientists’ significant experience with such analyses is what gives them their edge — we may take practice in considering the effects of people’s backgrounds, thoughts, and behaviors for granted, but it’s not something most people spend a lot of time doing! Of course, the particulars of the data you work with will depend on the position, but I feel comfortable saying that in many jobs out there, much of what you do will entail investigating and predicting what some type of people do — be it buyers, sellers, viewers, users, interactors, drivers, riders, whomever — and how it varies depending on who they are (e.g. demographics, psychometrics), what they’ve done (e.g. past behavior features), and what they think.

More specifically, experience with data about people as commonly gained in the social sciences comes in handy in three particular regards. First and most directly, having thought a lot about demographic and psychological differences in outcomes really helps with designing methodologically rigorous analyses that investigate such differences. This is particularly true in the case of those who have received formal training in research design during their schoolwork, capable of fluently identifying and explaining mediators; moderators; threats to internal, external, construct, criterion validity; and so on, perhaps even able to design and analyze solid randomized experiments. Your experience considering and accounting for such elements will help design rigorous analyses that cleanly measure what they proclaim to measure.

Building directly on this, secondly, coming from social science can be great help in modeling. Having a sense of the range of demographic and psychometric factors that can affect a particular variable as well as how such factors can interact with each other is a valuable skill to have when it comes to constructing regression and classification models. Such intuition can be of use in feature transformation (e.g. knowing certain variables, like income, are usually skewed) and feature selection (when more systematic methods aren’t ideal or feasible), and can also help with specification of more complex models (e.g. hierarchical linear models, structural equation models) as the structure of the data deems fit.

Lastly, a less obvious way a social science background helps in data science is with regard to feature engineering. Those from social science have no doubt encountered various theories that aim to explain particular phenomena or behavior, often with an accompanying diagram showing how certain constructs directly or indirectly lead to an outcome. Such theories can be useful frameworks for feature engineering, especially if one has experience coming up with clever proxy variables for otherwise difficult to measure variables. For example, the integrative model of behavioral prediction [2] posits that intent to perform a behavior can is predicted by attitudes (how one feels about the behavior), perceived norms (perceptions of how ‘normal’ a behavior is), and self-efficacy (whether one feels they can perform the behavior). In the context of, say, predicting whether user A will listen to song X, existing consumption data and relevant metadata could be used to generate features that approximate such constructs — for example, user A’s attitude toward song X could be represented by user A’s consumption of songs with high metadata similarity to song X; user A’s norms about listening to song X could be represented by the extent to which users with listening histories similar to user A have listened to song X; and self-efficacy could be represented by the variety of different artists, genres, and nationalities of songs user A has listened to.

All of the above points are especially true if you’re fortunate enough to work in a field where the research is directly applicable to what you do, as is the case for me. Having conducted significant research on media preferences and effects, I feel exceptionally comfortable developing models aiming to predict media consumption at various levels. But even if your field of study doesn’t perfectly project onto your industry, you will over time observe parallels in methodological approaches that you may wish to look into. For example, you might have a lot of experience applying text mining techniques to documents, but then you might realize that such techniques can be reasonably applied in any context where the data can be considered a body of text, not just documents — whether user metadata, song metadata, consumption history, or what not. Keeping an open mind about how certain frameworks from a field you’re familiar with could be applied elsewhere is critical.

The Nuts and Bolts: Key Technical Skills

But all your schooling and wit won’t take you anywhere in data science if you don’t have a certain set of key technical skills. Especially with all the different ‘specializations’ you see touted on various jobs pages — experimentation, inference, machine learning, visualization, and so on —it can be difficult to get a sense of what tools you really want to have a good grasp. Certain jobs will definitely call for heavier application of certain techniques, but I do feel there is a ‘core’ toolkit that all data scientists are expected to be familiar with.

Spoiler alert: Practically all such skills involve some elements of coding. I understand the idea of coding can be daunting for some, but the bottom line is that if you want to be in data science, you have to get comfortable with coding and the trials and tribulations that come with it. Take a course, get a certificate, do whatever you need to do to get comfortable with the basics. A good place to start might be to Google for a tutorial on a data project that sounds interesting and go from there. No matter where you start, just know that Googling problems and troubleshooting with the help of StackOverflow et al, where other people have run into the same issue, is a timeless tradition that even professionals do all the time, nothing to be frustrated by. So let’s dig into what I think are the key skills and concepts you should be familiar with.

• SQL: This really tripped me when I was on the job market — none of my classes taught it, and it’s something I’ve also heard from industry folks that people coming straight from academia often lack skills with. Sure piecing together and transforming data may sound boring, but the number of data-related job descriptions that call for SQL is really high. Plus, on a day to day level, there can often be a lot of quick and dirty analyses that can be done directly in SQL to save time. Yet at the same time, SQL is one of those things that it’s hard to gain functional practice with without some kind of live database that you can practice on. In that case, try to get comfortable with basic core dataframe operations like selecting, joining, filtering, and aggregating in other contexts like Pandas in Python or dplyr in R. With the right libraries, you may even be able to practice SQL in Python/Pandas [3] or in R [4].

• Python: Also, I love R as much as the next person, but Python is really the lingua franca of DS languages as far as I can tell. Python is far more versatile than R outside of the rare analysis where a particular function you need isn’t as robustly implemented in Python (and just forget about SPSS/Stata/SAS). When it comes to serious data science work, especially that which needs to be put into production (i.e. run regularly in an automated, robust, stable manner), Python is second to none at least at time of writing in 2021.

• Spark: At least a basic familiarity with the Spark framework — how distributed computing, compute clusters, and so on work — is really critical when it comes to working with big datasets. It’s easy to think 32GB or even 16GB of RAM on some local workstation is enough for chunky datasets, but you will undoubtedly encounter a situation where working on a single machine with a massive dataset loaded dreadfully slowly into memory is simply not feasible. This is another one of those cases where it’s difficult to gain practice with the tool without a live deployment, but at least be familiar with the underlying concepts if you can.

• Statistics, hypothesis testing: People love to ask to compare data and tell them whether they’re significantly different, often without really understanding what statistical significance means. As the data expert, you yourself definitely want to be familiar with hypothesis testing and basic statistics to know which tests are appropriate to use in which scenarios as well as to be able to competently explain the outcomes of statistical tests to others.

• Classification/regression: I feel like people often like to focus on classification over regression for some reason. Maybe it sounds a little glitzier when you can talk about precision and recall and some model can identify whether something is X or not, but let’s be real here: a lot of times your model-building will be for the purpose of predicting values, not labels, thus the importance of regression. Relatedly…

• ‘Core’ machine learning algorithms: No matter the type of data scientist you are, I’d say these days some familiarity with the mechanics of the ‘core canon’ of machine learning algorithms like LASSO, random forest, SVM, boosting, etc. is necessary to not be outdated. Regardless of whether you’re looking to do classification or regression, you will sound outdated if the only techniques in your toolbox are multiple regression and logistic regression. Such nonparametric algorithms are great when you don’t care as much about understanding the underlying relationships between the predictors and the outcome and just want to try to maximize predictive performance. That said…

• Linear models: Although such black box algorithms are great, when you want to make inferences about patterns in the data, linear models are still arguably the most useful. So having some fundamentals in techniques like multiple regression, multilevel regression, logistic regression, and so on is important. Plus, you’d be surprised just how far relatively simple methods like multiple regression and logistic regression can get you in a lot of cases, at times performing even better than the new-fangled methods mentioned above.

• Basic dimension reduction, clustering : Both of these methods are useful for visualization at the very least. Dimension reduction is also useful when you don’t want too many interrelated predictors in a model, and similarly, clustering can also help ascertain patterns in the data so as to be able to categorically label certain chunks of data as falling into a similar category. Principal components analysis and exploratory factor analysis are good places to start for dimension reduction, and k-means and DBSCAN for clustering.

• Clean coding technique: Lastly, this was another one I struggled with because I never had any proper computer science training or experience: it’s important to be able to write clean code. When you write code for work, you have to remember that you are not the only one who will be reading the code. You can’t just use acronyms and abbreviations that make sense to only you all the time, and you have to have some consistent and logical structure to the way to structure your code, name variables, and space out text. Get used to ordering your analyses in a clean, straightforward flow; naming variables very clearly (as opposed to ‘df,’ ‘test_df,’ ‘df_1,’ or the like); and perhaps checking out code style guides like PEP 8.

Book Recs

A big part of being a data scientist is constantly learning new concepts and methods. You’d quickly go broke if you took a paid course for everything you have to learn, so often times your go-to resources tend to be online tutorials and the like. I find books to be a happy medium between courses and tutorials, effectively a set of tutorials structured like a course. Here are a few books relating to the above I found useful as I prepared to enter data science, both to pick up new material and to brush up on concepts:

• SQL in 10 Minutes (Forta): The first book I used to start learning SQL, providing a series of lessons with a nice difficulty progression.

• SQL Practice Problems (Vasilik): Once I picked up the basics with the Forta book above, I used this book to practice writing SQL queries.

• An Introduction to Statistical Learning with Applications in R (James et al.): Yes, it’s an R book, but I think this book presents some of the most accessible explanations of common machine learning algorithms.

• Experimental and Quasi-Experimental Designs for Generalized Causal Inference (Shadish et al.): Not a technical book, but a very good, if dry, tome on designing methodologically rigorous studies.

Some General Parting Advice

Okay, so far, I’ve given you an idea of how your social science background can be valuable in data science, outlined the tools and concepts you want to be familiar with to help make a successful transition to data science, and listed a few books you might find useful as you get started. In closing, I have some general advice for social scientists aspiring to enter data science that felt a little too broad to include in the previous sections.

First, no matter what field you choose to go into to as a data scientist, make sure you understand the field as best as you can. It can be assumed that many of the candidates for any given data science position have hugely overlapping technical skillsets, and clear domain knowledge pertaining to the industry at hand can push you above the top compared to other candidates. Think about it from the manager’s perspective: when everyone has a roughly similar skillset, wouldn’t you rather hire someone who understands the concepts and metrics on which those skills would be applied?

Second, never, ever be afraid to think ‘weirdly’. This is another area where I think coming from social science really helps, because there’s often so many theories that aim to explain the mechanism driving various outcomes and the process of innovating often entails making a ‘weird’ connection between two things that hasn’t been made before. If you feel like there’s connection between two things, a method from one context you think could be applied in another, don’t hesitate to look into it or bounce the thought off of others. I don’t mind airing the occasional thought that makes me sound like a bit of a lunatic, because I know that every once in a while, one of those comments will get at something really insightful or novel.

And lastly, no matter where you go and what you do, pursue your passions. There’s a Ralph Waldo Emerson quote that’s long been a favorite of mine:

The voyage of the best ship is a zigzag line of a hundred tacks. See the line from a sufficient distance, and it straightens itself to the average tendency. Your genuine action will explain itself, and will explain your other genuine actions. Your conformity explains nothing. Act singly, and what you have already done singly will justify you now. [5]

I never grew up wanting to be a data scientist, but at every step I pursued my passions, drawing the zigs and zags of my ship through my genuine actions. Looking back now, I can see them straightening to their average tendency, enabling me to see how everything I’ve done has brought me to where I am now. I hope that as you progress in your career, you pursue your passions and draw the zig zags of your own best ship.

Signing Off

That’s all for me, folks! I hope this advice is useful to you, I really tried my best to encapsulate here all I wish I’d known when I was on the job market as a social scientist aspiring to be a data scientist. Best of luck in your journey toward becoming a data scientist!

References

[1] NCES, Most Popular Majors (2021), https://nces.ed.gov/fastfacts/display.asp?id=37

[2] M. Fishbein, A Reasoned Action Approach to Health Promotion (2008), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2603050/

[3] A. Bivona, How to Use SQL in Pandas (2020), https://towardsdatascience.com/how-to-use-sql-in-pandas-62d8a0f6341

[4] G. Grothendieck, sqldf: Manipulate R Data Frames Using SQL (2017), https://cran.r-project.org/web/packages/sqldf/index.html

[5] R. W. Emerson, Self-Reliance (1841), https://www.gutenberg.org/cache/epub/16643/pg16643-images.html#SELF-RELIANCE

Enjoyed this post but even a free subscription feels like too much commitment? It’s okay, we can keep this a situationship for now, but feel free to:

Grab Me A Drink 🥂

Otherwise, subscribe for free (or paid!):

REISSUE is a limited series on SeePlayLive where I transition over writing initially published elsewhere (e.g. Medium) that I think could be of interest to my Substack readers. Enjoy!

NOTE: This piece was originally published in 2024, prior to adjustments to various visa programs implemented in 2025-26 by the US government (e.g. $100K H-1B fee, etc.). While the broad strokes of topics discussed below remain generally accurate, I emphasize as I will again below that I AM NOT A LAWYER, and you are encouraged to seek an immigration attorney for formal legal advice.

You’re an American data hiring manager. After months of interviewing, you’ve just made an exciting new hire. You’re informed — either by HR or the hire themselves — the new hire needs company involvement to sort out their blahblahblah work authorization with the US government.

That blahblahblah is some esoteric and unfamiliar set of letters and numbers. Maybe a C and a P and a T for an intern, an H and a 1 and a B for a mid-level hire, or maybe that rockstar hire is insisting you support their case for E and B and 2. CPT… Captain? Is H1B a chemical or a droid name? There’s a successor to EB Games now? you quietly wonder to yourself. WTF? are the letters coming to your mind in response.

You don’t know what these numbers and letters mean, all you know with international hires is:

1. HR and/or Legal will take care of it, whatever “it” actually entails.

2. It’ll probably take a bit of time for all the paperwork to settle so the new hire can start.

Most American hiring managers have little motivation to be personally familiar with the trials, tribulations, and technicalities of US immigration. They, or most American citizens in general, have no reason to know that the average Joe non-American can’t “just get” a green card, citizenship, or job in the US with the snap of a finger. Short of marriage to a non-American or maybe being very, very good friends with someone going through the immigration wringer, most US citizens will never have any reason to learn what all these letters mean.

I firmly believe that given the supply-demand disparities in the data science job market both generally and in the US ([1], [2]), it is beneficial for hiring managers to understand the basic processes and terminologies of US immigration to optimize planning, enhance situational awareness, and increase empathy in hiring pipelines and general managerial functions. I wrote this article to try to cover those basics for any manager in the data realm who’s interested.

Before I get started, please allow my inner teenager to find juvenile amusement in this apt opportunity to use the acronym IANAL — I Am Not A Lawyer. This article is NOT meant to be legal advice, but instead an accessible overview of what that intern, analyst, data scientist, data engineer, or the like you’re hiring is going through so you can plan accordingly and don’t feel like they’re throwing around random characters. Please alt-F4 or cmd-Q out of your browser and seek the advice of an immigration attorney if it’s more formal immigration advice you’re looking for rather than my snarkily charming and accessible two-cents.

With that out of the way, let’s dig in. I’ve loosely divided the content below by seniority of the target hire. I make no claims as to be providing any kind of comprehensive overview— again, IANAL, hit up your HR/legal department or an immigration lawyer for the full deets — and there are niche-case specifics I omit and some processes/terms I significantly simplify on purpose, but what I mention here should be relevant to the vast majority of international hire cases you’ll encounter*. The title is, after all, Immigration 101, not Immigration 499: Special Topics.

*I would love to be able to speak to international immigration processes more broadly, but my knowledge is limited to the US immigration system, hence the focus on immigration processes in the US here.

Interns — F-1/Curricular Practical Training (CPT)

An international student at a US university that you hire as an intern will likely need some support for their Curricular Practical Training, a.k.a. CPT, authorization. F-1 is the student status international students generally come to the US under, and CPT is the program intended to give F-1 students hands-on work experience in a job related to their field. The intern needs to bring to their university’s international student office (abbreviated ISO henceforth) an offer letter for a job related to their major to get the ISO’s authorization to work for you; this ISO authorization can happen pretty swiftly, within a day or two to a week with standard processing times varying by university. CPT will also often entail the student registering for some kind of internship course at their university, the requirements of which vary by institution and field — so yes, this means if the internship is unpaid, an international student can end up going into the red just to be able to even sign up to work for you (but you ARE paying your intern, right???).

Things hires in this phase might stress about: Getting the offer letter worded/properly formatted for their ISO, getting quick-turnaround CPT authorization from their ISO depending on hiring/school timeline, clarifying specific start/end dates to provide to ISO for CPT authorization, asking you for some kind of periodic review or such paperwork for their internship course.

Recent Graduates — Optional Practical Training (OPT), STEM-OPT

An often subsequently used “sister” program to CPT is Optional Practical Training (OPT). While sometimes used prior to graduation, OPT is generally thought of as work authorization for immediately after the student graduates. OPT can be hairier to handle than CPT given that instead of being solely authorized by the university ISO, the student must send off the OPT authorization from their ISO to the federal government (USCIS) to receive their work permit — and the nature of these USCIS processes is that they can move slower the standard expected processing times depending on caseload, political climate, the alignment of the planets, and the way the wind blows (i.e. for little clear reason). This is why you sometimes hear of recent graduates with standing offers pushing back their start dates bit by bit as they wait for their OPT work permit to arrive.

Closer to the conclusion of their initial 1-year OPT period, students with degrees in fields deemed STEM (science, technology, engineering, math) by the US government are also eligible to apply for a two-year extension to their OPT period, called STEM OPT. This means that students with STEM degrees can have up to a total of three years work authorization after completing their degree. STEM OPT requires the employer to be E-verified, which can be a bit of a complication depending on your company.

Things hires in this phase might stress about: Submitting paperwork for and waiting for their OPT authorization to arrive, hoping they don’t lose their job offer if the OPT authorization takes a long time to arrive, figuring out whether the company has E-Verify to make STEM OPT possible, submitting paperwork for and waiting for STEM OPT authorization to arrive.

Mid-career Professionals — H-1B, EB-2

Mid-career hires and graduates seeking to continue their employment beyond the 1 or 3 year OPT/STEM period will generally seek employer sponsorship for the famous H-1B visa for specialty occupations. Most simply put, H-1 status lets a person holding a license/certification in their domain, a bachelor’s degree or higher, or comparable experience work for whatever company sponsors their H-1. The trick with H-1 is that demand exceeds supply, so there is an annual lottery to determine who is eligible to even submit an application. Indeed, only a fraction of H-1 lottery entrants are selected each year [3] — so yes, it is entirely possible for a perfectly qualified individual to be unable to get an H-1 visa due to dumb luck.

An employee that’s more seasoned and is transferring an H-1 they got through the lottery at a different company — and thus can be hired by your company without dealing with the lottery — may be tired of continuously extending their H-1 and seek your company’s sponsorship for permanent residency, a.k.a. a “green card.” A common way for more experienced professionals to get their green card is via the EB-2 status, which grants permanent residency to individuals with either 1) an advanced degree, 2) a bachelor’s degree with 5+ years of experience, or 3) exceptional ability in their field. The most common route to take is one of the first two, and both of these routes require sponsorship from the employer and obtaining something called a labor certification from the US government (Department of Labor), which effectively checks to make sure that the employer did their due diligence to try to hire a domestic professional and is not underpaying the foreign professional relative to the prevailing wage for their occupation.

Things hires in this phase might stress about: How long they need to work for the company under OPT/STEM OPT before the company is willing to file H-1B paperwork, trying again after not being selected in the H-1B lottery the first time, how long they need to work for the company under H-1B before the company is willing to file green card paperwork, assembling documentation for green card application, processing time for labor certification, priority date arrival for green card application, processing time for green card application, promotion effects on visa or green card case.

Exception Cases — O-1 and EB-2 NIW/EB-1

Professionals with exceptional achievement in their field may opt to skip putting their fate in the hands of chance in the H-1B lottery and request company sponsorship for an O-1 visa. The O-1 visa is for “individuals with extraordinary ability in the sciences, education, business, or athletics,” with “a level of expertise indicating that you are one of the small percentage who have arisen to the very top of the field” [4]. In support of the O-1 application, the company (working with the hire) must provide documentation of evidence relating to the field of expertise such as internationally recognized awards, scientific publications, participation in panels or as a judge of others work, and the like.

The O-1 is available without lottery drama to anyone who passes the high bar, but it is ultimately still a temporary visa just like an H-1B. So hires who are tired of the visa drama may also request green card support. They may go the standard EB-2 route as described in the previous section, but if they’re on O-1, they may also qualify for the similarly ability-based Extraordinary Ability EB-1 or National Interest Waiver (NIW) EB-2 green card paths that don’t require employer sponsorship (yes, with the latter you literally have to prove it’s in the national interest to give you a green card). Not needing employer sponsorship means they’re less tied to the company and have less to lose by leaving, but it also means you won’t have to deal with as much hoopla, maybe a couple of administrative assists with employment verifications and job descriptions and such documentation here and there.

A Few Final Notes, and Concluding Remarks

So “that’s it!” If it all sounds hairy, that’s because it is! And for almost all of your international hires, there is a certain degree of immigration anxiety constantly looming over their heads to various degrees. This is especially true because most of the time, if they were to lose their job, they’d only have something like 60 days to find a new job or self-deport lest they accrue unlawful presence! And if you accrue too much unlawful presence and leave the US, you can be banned from re-entering the country! A few more quick bits before we go:

• Visa vs. status: Visas and status are highly correlated but not the exact same. The visa is the stamp in the passport, which indicates to Border Control when an individual enters the US that they hold a particular status. A person already in the US can adjust their status (e.g. from F-1 student to H-1 worker), such that while they have valid status, they no longer have a visa in their passport linked to their current status. This means the next time they leave the country, they have to go to the US Embassy to get the visa stamp in order to re-enter the country.

• Visa vs. immigrant visa: So when people say “visa” they generally mean one of the temporary, non-green-card-producing, nonimmigrant visas like F-1, H-1, and O-1. Technically speaking, EB-1 and EB-2 are also visas, but they are immigrant visas, i.e. they grant you a green card; for this reason, the use of the term ‘visa’ is constrained to the F/H/O visas in practice. In the framework I highlight in the above bullet, the idea is that if you apply for EB-1/2 abroad, you get a corresponding stamp in your passport to enter the country and are immediately considered a green card holder, but you can also adjust to EB-1/2 status (from O-1 status, for example) while you’re in the country.

• Different numbers after the letter: Each letter-number combination notes a different type of status and they can all be entirely different, but the most common variants you’re likely to encounter are dependent statuses. Dependents of F-1 holders get F-2s, dependents of H-1 holders get H-4s, and dependents of O-1 holders get O-3s.

And if you’ve gotten this far, thank you for reading. Clearly you care about what your employee is going through. I hope you agree that given the supply-demand dynamics in the US data professional labor market, a basic knowledge of how these immigration processes work can be beneficial to data team planning and operations. Lastly, while broadly current to the best of my knowledge at time of writing, the nitty-gritty of these matters can and do change as laws and administrations change, so another reason you should talk to HR/Legal for the final word — except this time, you can walk in informed and knowing what the various letters and numbers mean!

[1] https://www.zdnet.com/article/employers-are-desperate-for-data-scientists-as-demand-booms/

[2] https://engineeringonline.ucr.edu/blog/data-scientist-shortage/

[3] https://www.forbes.com/sites/stuartanderson/2023/05/01/immigration-service-likely-to-change-h-1b-visa-lottery/?sh=6b0036f42868

[4] https://www.uscis.gov/working-in-the-united-states/temporary-workers/o-1-visa-individuals-with-extraordinary-ability-or-achievement

Enjoyed this post but even a free subscription feels like too much commitment? It’s okay, we can keep this a situationship for now, but it’d be great if you could:

Grab Me A Drink! 🍸

Otherwise, subscribe for free (or paid!):

Recently, one of my MBA classmates contacted me out of nowhere, asking me for my input on navigating the job market.

Now, while I do take pride in elements of my background, I have always acknowledged that on the years-of-experience front, my profile is nothing special. I didn’t finish my education and start a full-time industry job until a little over 3 months before I turned 27; meaning I easily have 4-5 years less experience for my age than people who started working straight after college. I’ve also really only held two jobs, meaning I’ve only gone through two job hunts all the way through to starting a new job (I did have a successful secondary job hunt that I will tell the story of another day).

I was upfront with the above disclaimers to my classmate as well as the caveat that norms can differ by domain before gathering my thoughts — and it’s those notes I want to share with you here today. Both of my main job hunts took place in less-than-ideal circumstances with various immigration related constraints, so perhaps out of necessity, I developed a system for applying to jobs that optimized my chances of an interview; a protocol that ultimately landed me both of my industry jobs — yes, the path to both of my jobs started with me straight-up emailing the hiring manager.

In this piece, I detail how you can be more than a row in a database some ATS scans through so you can optimize your chances of getting to talk to a real person about your experience and background. In my opinion, getting to the interview can be just as hard as if not harder than succeeding in the interview itself in many ways, and just managing to get yourself to the interview phase means you’re ahead of hundreds of applicants a lot of the time.

So what is my job hunt process? Again, the applicability of some of these tidbits may vary depending on your particular industry, but let’s dive in.

1) I focus on recent job listings.

I’d always known it was good practice to jump on job listings as quickly as possible, but I really came to appreciate this when I had to hire interns. No matter how exceptional you are, if you submit your application after the hiring team has decided they have a sufficient pool of interesting candidates, your odds of getting your application seen rapidly diminish. That said, some of the following steps should help get eyeballs on your resume even if you throw your hat in the ring late.

2) I don’t bother with cover letters.

I don’t think anyone reads these. I’ve even heard recruiters and hiring managers say they don’t read these. I only gave them a quick look when I was hiring interns since while everyone had a resume, most didn’t have a cover letter, so focusing on resumes felt more fair. Your energy is better spent elsewhere, as I shall detail further below.

3) I use LinkedIn Premium to look up hiring managers and recruiters for positions I apply to.

When I'm on the job hunt is pretty much the only time I pay for LinkedIn Premium, and I use that to look up the people I think I might be the hiring manager or recruiter for any position I apply to; you can actually do most of this sleuthing with LinkedIn Free, but Premium opens up InMail options as I discuss below. Your target job listing is for Manager of BlahBlah at CoolCo? I’ll look up Director BlahBlah CoolCo on LinkedIn. The listing is for Director of BlahBlah at CoolCo? Let’s search for Vice President BlahBlah CoolCo, etc. At the very least you should be able to find someone who is a recruiter at the company. It’s the best when your life is made easier by the hiring manager themselves posting about an opening on their LinkedIn page, then you don’t have to wonder who it is.

4) If you can figure out the email address format for a company, you can guess the email address of most people who work there.

Usually, you can find email addresses rolling around on the internet that give you an idea of what a company’s email address format is, i.e. is it first_last, first.last, firstinitialLastname, and so on. There are even websites like RocketReach that will help you with this endeavor. Once you have knowledge of a company’s email address format, figuring out the email address for hiring managers and recruiters becomes easy as cake.

5) Instead of a cover letter uploaded with my application, I take the knowledge from 3 and 4 and write emails directly to recruiters and/or hiring managers after I apply for the position.

I always email who I think might be the recruiter or hiring manager for a position after I submit my application, providing a brief intro about myself and my interest in the position alongside job posting ID and application ID as appropriate. If I can't figure out who the hiring manager for a position is, I generally try to email someone I think is either their boss, a colleague of theirs at the same level, or someone on their team (in order of highest to lowest preference). If I have to pick between emailing the recruiter and the hiring manager, I prefer to contact the hiring manager, but both never hurts. Though it can sometimes be hard to pin down the single specific hiring manager or recruiter for a position, I've found if you're a good fit for a position, as long as you connect with someone in roughly the right direction, they often help get your resume to the right person.

6) If email doesn't seem to work no matter what you try, then you fall back to using your LinkedIn InMail credit to do the direct outreach.

I 100% prefer email over LinkedIn whenever possible, to the extent that I have shamelessly bcc’d 4-6 different combinations of first name, first initial, last name, and last initial and in the hopes one of them wouldn’t bounce — hilarious, I know. But if that really, really doesn’t work, you can always fall back to LinkedIn InMails. It’s also a bonus if your profile is industry- and function-aligned enough for whoever you need to message to accept your connection request, then you can message them without using InMail credits!

Enjoyed this post but even a free subscription feels like too much commitment? It’s okay, we can keep this a situationship for now, but I’d be super grateful if you could:

Grab Me A Drink 🥂

Otherwise, subscribe for free (or paid!):

A common thread you may notice through each of the principles I’m about to discuss is the idea that AI use must be a dialogue, not a dictation. You must be engaged, even if not expert, with regard to the topic domain in which you are trying to use it. You cannot just blindly follow its outputs as dicta — you must know how to push back. AI cannot be a substitute to your actual intelligence, it must be an augment and accelerator.

As usual, I don’t claim anything here to be some kind of stone tablet divine tenets. And obviously, I’m focused here more on realistic, everyday considerations - like, it should be obvious I refuse to use AI to develop weapons that help me take over the world without me explicitly detailing so below. Crazy what-ifs aside, I highly encourage anyone who also uses AI frequently in various everyday contexts to develop a personal code of AI use that sets out what you will and will not use AI for. Even if at present there’s no need for you to draw a hard boundary on the matter in a certain context, there may come a time where you are pressured to do so, and only by having really thought about it in advance will you be able to make a decision you can live with.

Writing: I do not use AI to write content for me.

I have never used AI to straight up generate AI content for me. I simply refuse. My writing skill is something I take pride in, a completely unprofitable, totally recreational craft that I’ve invested countless hours into. I have a particular sense of written voice I’ve cultivated over the years that I’ve done my damndest to have parallel my spoken voice, and I don’t trust AI to be able to capture that in any substantive way right off the bat. And if my writing doesn’t reflect my own voice, my own take on things, then why the hell would anyone read my shit? As far as I care, using AI to generate significant chunks of text while claiming to be a writer is like depending on Grammarly while claiming to be a copy editor; you should probably find a different profession.

Counterpoint: I will happily let AI shop ideas with me or edit my writing for me.

I’ve discussed in other pieces (such as that linked below) how AI can be so great in helping you do historical and technical research to make whatever you’re writing sounds authentic to the period and domain, but it can serve other functions as well. While I will not let AI write stuff for me, I love using it basically as a friend I can bug at anytime, as many times I want to bounce different ideas. There are limits to how much you can talk through or iterate an idea with a real friend, especially if you spent a lot of time on the same idea; no such limitations exist with AI. Also, while most smaller pieces never even make it to this step — e.g. my Substack pieces are never AI post-processed in AI — for larger pieces such as screenplays or novels such, I love going through each scene or chapter with AI one by one and tightening them one.

Code: I do not use AI to code anything larger than a single script.

The very first time I got Cursor, a modern AI-enabled IDE, to work on my business idea, I thought I could just tell it what capabilities I wanted the software to have and have it basically write a first draft of the code into a GitHub repository. While yes, this is feasible, I quickly learned that I had no idea what the fuck was going on. All I saw was a bunch of different code in a bunch of different places, some of which seemed capable of doing the things I’d asked, but I had zero intuition for how the entire system was structured and when/where different code was executed. That’s when I took a step back and decided that I’m not gonna use AI to write me anything larger than a specific script or, more frequently, a function. Once I review and understand that small building block it generates, I will then proceed to further collaborate with AI generate more building blocks and stack them all together in an intentional, thought out manner.

Counterpoint: I will happily let AI debug code for me.

I personally consider debugging code to be hell, and nothing induces more anxiety in me than when a work-critical piece of code bugs out and I am the one expected to fix it. Even if I ultimately resolve the issue just fine — which is the case pretty much every time — I just do not need the cortisol hit in my life, nor does that hit help me debug at all in the first place. AI has been an absolute godsend in this regard, oftentimes not only providing insights on how to fix the issue but straight up editing the code to directly resolve the issue. When you do this however, testing to make sure the fix fully works as intended is key.

Social: I do not let AI tell me what to say to someone.

I refuse to let AI tell me what the best message to send someone might be. Hard pass. This is especially true for dating apps — where the idea of leading with an AI composed pickup line makes me shudder — but it also holds true for other contexts, to the degree that I also refuse to use those auto-reply suggestions LinkedIn messages give you. I have a genuine fear that, the ways things are going, AI trained on pre-existing corpora will end up predefining and ‘boxing in’ how humans not only communicate but also feel, and I refuse to play a part in that. We have to preserve that human entropy, the beautifully ordered chaos in how we communicate and feel, and not let AI turn all our interactions and emotions into something a monochromizing rock tumbler of existence might spit out.

Counterpoint: I will happily let AI help me figure out how or when to say it.

While I won’t let AI tell me what to say to someone, I am more than happy to use it to figure out what the best way to say something to someone is. You can phrase the same sentiment and intent a million different ways, and how you phrase something can absolutely impact the outcome. I can sound a little clinical or overexplanatory in my writing, and AI sometimes helps me avoid that tonal speed bump in my written communications. Boiling it down to the basics, you know how I mentioned above that AI is like a friend you can bounce story ideas off of any time of day, as much as you want? To be super reductivist: if you’ve ever messaged your Council of Friends with a draft text intended for someone you like asking, “Guys/girls, does this sound okay: ’___’?” — AI is the friend you can ask that question to any time of day, as much as you want.

Enjoyed this post but even a free subscription feels like too much commitment? It’s okay, we can keep this a situationship for now, but it’d be great if you could:

Buy Me A Drink! 🥂

Otherwise, subscribe for free (or paid!):

So I’ve decided I want to start a business. Two actually.

One of them is more conventional and leans into my job-related skills - let’s call this the data biz. The other is less conventional and would involve leaning into marketing and selling in ways I’ve never done before - I’ll refer to this as the item biz.

I’m gonna focus on the data biz to start - it’s a little more serious, and has a lot more long term potential. I’ve had a lot of thoughts and questions come up already as I go about the very, very early stages of exploring my entrepreneurial ambitions, and I figure I’ll only have more come up as I continue down this path. So, I thought, why not start writing them down?

I figure someone might find these notes useful down the line. And much as was the case with writing and my starting this Substack, part of the intent behind the Entrepreneur Chronicles series is to keep myself accountable and motivated in pushing the business idea forward.

Here are some of the (very basic) things I’ve already started thinking about:

How much should I talk about it?

Other than conversations with a few select close friends and family, I’ve decided I’m gonna keep details on the data biz idea on the down-low for the time being. I’m not yet gonna go so far as to put Part-Time at Stealth Startup on my LinkedIn with that funny little F-115 icon, but I’ve been very careful about where I openly discuss and mention the business idea. Even putting aside paranoia about stolen ideas, part of my excuse for the discreetness is also a partial belief in the idea that talking about doing something gives your brain a similar sensation as actually doing it without having done it.

Can I deduct business-related expenses?

This I actually think I know the answer to. If I understand correctly (I am not an account, please do your own research), up to $5,000 in “startup costs” and separate $5,000 in organizational costs related to the establishment of your business entity can be deducted to lower your taxable income. However, there does seem to be some conflicting information on whether equipment can be included in startup costs, specifically starting tax year 2026. Sounds like I’ll have to talk to an accountant come tax season.

When should I ‘formalize’ it with an LLC and bank account?

This is what I’m struggling with the most. At least for now, any work I do related to the data biz feels just feels like a very expensive hobby. But at the same time, it costs money establish an LLC and maintain a business checking account. Plus, costs incurred aside, I’d feel like too much of a poser if I had an LLC and business bank account but wasn’t doing anything with it. I’ve decided that the milestone at which I will move to establish an LLC and business checking account is when I actually have something I could take to a potential client to work with.

How is a B2B business supposed to get customers?

So while my item biz is B2C, my data biz is B2B. Though obviously B2C comes with its own set of intricacies and challenges, I’ve always struggled to wrap my head around how a B2B firm is supposed to go about acquiring clients. In my case, this becomes most immediately relevant when I start looking for potential partners for testing my B2B product, partners who could later become clients. Seems like early forays could be awfully depressing to the level of handing out flyers on the street - most people ignore you and treat you like a nuisance. Guess I’ll have to find out the hard way, keep an eye out for this question to return in the future.

Enjoyed this post but even a free subscription feels like too much commitment? It’s okay, we can keep this a situationship for now, but I’d be super grateful if you spare a few bucks to:

Grab Me A Drink! 🥂

Otherwise, subscribe for free (or paid!):

Not gonna lie, one of the earliest motivators of my decision to do an online MBA was feeling like my salary should be higher when I started my job. Despite the usual back-and-forth, HR hadn’t been willing to budge on the salary and had only added a small signing bonus to sweeten the deal.

So once I was onboarded, I looked at all the various employee benefits available to make the most of the resources offered to me. The one that immediately caught my attention was a $9,000/year tuition reimbursement benefit. I decided to construe that as the company offering to pay me an extra $9,000 a year to improve myself, and I sought out ways to make the most of it.

Let’s be real here: $9K a year will not go far these days with traditional degree programs, which can cost tens to hundreds of thousands of dollars. $9K a year would be a drop in the bucket in such cases, and I was not about to go into student loan debt for the first time just to use my tuition reimbursement benefits.

I knew online programs, even affordable ones at that, existed, offered by all sorts of universities around the US. But I was determined to not rock the education section of my resume with a degree that looked out of place next to the others - so any online master’s degree I did had to be from a reputable R1 university with a recognizable brand.

Perhaps unsurprisingly, I found during my research that balancing cost and reputation is tricky when it comes to online master’s programs; plenty prestigious universities’ programs demanded a second mortgage, while many affordable programs were offered by universities I had never heard of. However, I was pleasantly surprised to learn that not all hope was lost. Various innovative engineering and business schools at major universities had put out reasonably priced degree programs intended to be disruptors in the space.

I wanted to publish my list of candidate programs from that search here in case anyone finds them useful. “Affordable” is ultimately a very relative term, but I considered a program affordable for the purposes of my search if it was $30,000 or less, i.e. could be covered by roughly ~3 years of my work tuition reimbursement. As for the reputation criteria, I mention the R1 categorization above, but I’m sure you can get a feel for the general vibe range I was open to when you look at the list.

By no means do I make any claim regarding the comprehensiveness of this list, there are surely great programs that I don’t include here. Also, while I did my best to make this piece accurate at time of publication, I may have missed details and programs are subject to change their requirements, costs, and details without notice - please do your own homework before submitting applications and the like. Lastly, any comments about curricula and course options are purely my personal opinion.

I hope this piece helps you get started in terms of directions you may wish to look for your online master’s degree. Programs within each section are listed alphabetically by university. Some of these programs/universities I’ve looked at more closely than others, and I’ve commented accordingly for certain entries.

Computer Science

Arizona State University - Online Master of Computer Science ($15,000)

Georgia Tech - Online Master of Science in Computer Science (~$7,000): To my understanding, one of the original disruptors in the online CS degree space. Seems like a pretty intense, rigorous program based on everything I’ve heard about it.

Purdue University - Online Master of Science in Computer and Information Technology (~$28,500)

University of Colorado at Boulder - Master of Science in Computer Science on Coursera (~$16,000): My theory as to why they emphasize their partnership with Coursera is that they’re trying to highlight that the program uses a cohesive, unified learning platform instead of a hodgepodge of different digital tools.

University of Illinois - Master of Computer Science Online (~$26,000)

University of Texas at Austin - Online Master of Science in Computer Science (~$10,000): I like the simple billing and curriculum structure of this program and its DS/AI siblings listed below - 10 courses for $10K (+fees).

Data Science, Machine Learning, and Artificial Intelligence

Georgia Tech - Online Master of Science in Analytics (~$12,000)

University of Colorado at Boulder - Master of Science in Data Science on Coursera (~$16,000)

University of Illinois - Master of Computer Science Online, Data Science/AI Tracks (~$26,000): Data Science and Artificial Intelligence are specific tracks within the MCS program listed in the above section.

University of Texas at Austin - Online Master of Science in Data Science/Master of Science in Artificial Intelligence (~$10,000)

Business Administration

There are a few other programs I contemplated including in this list (Iowa, Nebraska, Auburn), but decided to leave out as by my calculations they cost more than $30,000 total.

Boston University - Online MBA (~$25,000): The MBA program I completed! Reputation and affordability aside, I liked that the curriculum was basically decided for me; though I geeked out over personalizing my curriculum in school, it was not something I wanted to deal with while employed full time.

University of Illinois - iMBA (~$26,500): The other MBA program I considered. It seems more personalizable based on your particular interests (e.g. focus areas in business analytics, digital marketing, mergers and acquisitions, etc.) than the BU program.

Enjoyed this post but even a free subscription feels like too much commitment? It’s okay, we can keep this a situationship for now, but it’d be great if you could:

Grab Me A Drink! 🍸

Otherwise, subscribe for free (or paid!):

In November 2025, I was invited to speak at the University of Arizona Eller College of Management’s Business Analytics and Data Leadership Forum. It was an absolute pleasure to get the opportunity to share what wisdom I had to offer with the next generation of data professionals. Because I wanted to share what I did with the students that day with a broader audience, presented here is an abridged version of my notes from that day, adapted to article form.

What is my background?

Before I get started doling out words of wisdom, I’d like to start with telling you a bit about my academic and professional background so you know where I’m coming from.

My undergraduate education was actually far from data science. I was a game design major in the film school at USC with a minor in music industry. Though I’d taken AP Calc in high school, I didn’t take a single math class during undergrad. Instead, I did have lots of super fun classes with seasoned industry professionals on the art and business behind film, games, and music. I also did my first master’s at USC, in communication management, during my senior year through a BA-MA program.

I then went onto do my PhD in communication at the University of Pennsylvania, where I focused my courses and research on media effects, media preferences, and quantitative methods. During my PhD studies, I also earned a master’s in statistics through The Wharton School and interned at Paramount Pictures.

How did I get started after school?

My first job as data scientist after my PhD was at Whip Media; some of you might be familiar with their mobile app TV Time, and my work at Whip focused heavily on streaming data. While I was at Whip, I also held a title as visiting researcher at USC, and through that affiliation I continued to publish media preferences research and present at various industry and academic conferences. After three years at Whip, I landed at Sony Pictures, where as part of the motion pictures group I lead the predictive modeling of opening box office revenues among various other projects. Once I started at Sony, I decided I wanted to make the most of the tuition reimbursement benefits I get, so I enrolled in the online MBA program through Boston University and finished that in December 2024.

What do I do at work?

Now, the unglamorous truth of being a data scientist regardless of industry is that you spend a lot of time cleaning up the data you’re going to use to do the so called Fun Stuff. You gotta ingest the data, transform it into the form you need it in, join them all together, and figure out how to deal with the missing pieces. Once you’ve done that, you usually start with looking at some basic descriptives like means, correlation, and so on. From there, you get to the model building, which can range from anything like regression and classification models, to the development of vector space models for representing things like movies, shows, or platforms in a multi-dimensional space. Lastly, once you do these tasks, you have to present your finding to both internal and external parties depending on the context.

What kind of data do I work with?

Without going into too much proprietary detail, the datasets I work with are largely what you’d expect; the magic isn’t in the raw data, it’s in what you can derive from them. I work with survey data, social media data, data on content, as well as data on audiences. Now that might not sound like a lot, but believe me it is, especially once you start processing, transforming, and feature engineering to suit your needs. In many regards, these datasets breakdown broadly into two categories: 1) Content-side data about what a given title is about and who or what is in it, and 2) audience side data about how people are thinking of, talking about, or otherwise engaging with a title.

What kind of projects do I work on?

So what’s an example of a project one might tackle with such data? Well, a good example would be the box office prediction model overhaul I’ve been working on. When I arrived at Sony, there was an opening box office prediction model that had been developed bit by bit in R by generations of data scientists before me. Just by virtue of its age, the codebase was messy and it used a fairly dated architecture, slowing everything down. As a result, I’ve been leading the effort to rewrite the model with a more modern architecture and algorithm in Python. These efforts have led to a prediction model that is faster and more accurate with a cleaner codebase and more comprehensive prediction-level insights. But of course, I work on a wide range of other projects involving analysis and modeling of content and audience data in line with various business needs.

What’s it like managing data scientists?

Most recently, I’ve been gaining managerial experience, and it’s a shockingly different ballgame. No one ever tells you this in advance – that or people do and you consider it a trite aphorism til you see for yourself – but managing the doing is a whole different can of worms compared to the doing itself. It calls for sufficient familiarity with the various components of the doing so as to be able to plan how you want the various components organized, in what order you want components completed given how long they’ll take, and how to spread these responsibilities across people. You also have to be good at picking really good people for your team. So yeah, managing is really a whole different set of skills than doing, but the former in my opinion inherently calls for deep experience with the latter.

Is ‘entertainment data science’ a monolithic single field?

Something I’ve come to feel over the years during my time in entertainment data science is that though it’s easy to think about entertainment media and data science and think it’s all the same, that’s really not true. Different contexts within the entertainment space call for different DS tools, processes, and questions. For example, my last job was in streaming data science while my current job is in theatrical data science. This, for one, meant that the datasets in my last job were often far bigger than the ones I encounter regularly in my current job, and the sheer size of those datasets called for the use of certain tools that I don’t need as much today. In addition, I’ve generally worked immediately before and within the post-release phase of the content life-cycle, but DS can be applied for example in pre-production contexts to help determine what content should be greenlit for production in the first place, or in production contexts to help optimize production planning and spending.

What should I know to become a data scientist?

If you want to become a data scientist, these tools and techniques and techniques in the picture above are what I think you should be familiar with. If it looks like a broad laundry of techniques, that’s because it is. I’m not trying to say you’re going to be using all of these techniques all the time at work; I’m saying that though you will no doubt develop a specialization over time – and data scientist titles often reflect this these days – there are certain techniques and topics that you are foundationally expected to be familiar with, and these are it.

I also want to draw your attention to the two I have starred and bolded and underlined, SQL and writing clean code. SQL’s hard to get practice with in school, but there are libraries for Python that will let you do dataframe operations with SQL if you want practice. And writing clean, accessible code is the kind of thing no one teaches you in school, but once you’re in industry, it’s essential because you will not be the only one looking at your code, your teammates will need to be able to interpret it and adapt it as needed. You can’t just name stuff df1 or test_df or main_data or have code that lacks a clear top to bottom flow; you have to name things clearly, structure things in a manner that makes sense, and use indentations wisely to keep things legible.

Lastly, these days, even if you don’t wanna get super technical with it, I really think people need to get familiar with the basics of working with LLMs. Like, you should know the basics of prompting, know what what system and user prompts are, the basics of how they work in the backend, and so on. I genuinely believe that the same way we might consider lm in R or read_csv in Pandas to be rudimentary and basic today, we will get to a point where people will consider working with an LLM that basic as well.

Do you have any book recommendations?

I find a few books especially helpful in transitioning to becoming a data scientist:

• An Introduction to Statistical Learning: With Applications in Python (James et al.)

• Statistical Learning from a Regression Perspective (Berk)

• SQL in 10 Minutes (Forta)

• SQL Practice Problems (Vasilik)

The first book is the Python version of the book I used in my first data mining class at Wharton, a super accessible intro to commonly used machine learning algorithms. The second book is from another one of my stats classes that I found super conceptually interesting. And the last two books are the books I taught myself SQL with when I was first entering the job market.

Thanks. Any real talk about being a data scientist you care to share?

To start, like I alluded to earlier, just know data processing is a big part of being a data scientist; so if the thought of doing a bunch of joins and pivots and filtering and such makes you cringe, well, better get used to it. And don’t think you’ll just gonna be hunting for insights all the time; data science goes from extracting insights to building robust data products pretty quickly as you gain experience.

Conversely, it’s important to remember you’re not gonna be doing complex modeling and analyses all the time as a data scientist; at another event, someone asked me if I make models for everything and I was like, “no, most of the time it’s just group by averages and correlations.” Relatedly, always remember to talk like a normal person; I still struggle with this myself sometimes, but you have to get really good at conveying complex ideas in an accessible way, and that also means knowing what not to bother conveying because the audience won’t care.

Lastly, know that a lot of the time, no one is expecting perfection, and you have to carefully tune your sense for how meticulous any given analysis needs to be, and let it go appropriately; it’s done when it’s due. But that all the more means you have to really know your stuff.

Give me some advice for while I’m still in school.

First, I think self-direction is key. You will never do the most innovative work if you’re always asking to be told what to do – you can’t expect others to tell you how to combine various threads, you have to figure out yourself how to combine threads that interest you in your own unique way.

Similarly, try to have a long-term view and plan out your path early on. Early on in my time at Penn, I knew I wanted finish in four years with both my PhD in communication and my masters in statistics. So I looked at all the program requirements and course schedule patterns available to me, optimized my course selection accordingly, and kept a running log of my personal course plan as I moved through the program. And I made it happen. Planning is half the battle to making things go your way.

Next, I think taking as many methods classes as you can is so important. It’s great to know theory to know what questions to ask and have a structured framework for thinking about topics, but methods knowledge stands on its own independent of theoretical domain. Try to have as wide an arsenal of methods - of different machine learning and statistical techniques - in your toolkit as you can.

And try to land an internship in your industry of choice while you’re in grad school. Yes, you will know people here and there that land a gig at some tech company right after grad school despite never having had an internship, but those will be the exceptions not the rule, and while it’s nice when you end up an exception, it’s one of those things you can’t just count on. I know for a fact it would’ve been far harder for me to land in the entertainment industry after grad school if it hadn’t been for my internship at Paramount.

This next point, I’ll lead into it with a minor anecdote. I remember once in grad school during the peak of my disillusionment with academia, I was venting to a bartender how I was tired of academics and she was like “how come?” and I said, “imagine someone read a bunch of books about bartending, maybe ran some surveys about bartending, then called themselves an expert in bartending. Academics can be like that.” By this logic, I implore you - if you consider yourself an aspiring data expert or media expert or entertainment data expert, please try your hand at making a data product or making a piece of media. With regard to entertainment data science, obviously you want data chops as a given, but if you mention on your resume you’ve made a short film or something like that, it will stand out a bit more as a marker of domain interest.

But yeah, short film, app, indie game, ML model, whatever, just throw yourself at making it - you will find that hands-on experience making the thing you choose to study the outcomes of will help you ask more refined and relevant questions.

Any final words of wisdom?

No matter what industry you choose to go into, make sure you know your industry. For example, in data science, lets be real here, to some extent everyone has very similar foundational skills. So beyond that, what really sets you apart is knowing the field and having very clear ideas for how various techniques can be applied to investigate questions that are relevant to the business. As one of my bosses once put it, anybody can run a regression or write a SQL query, but what really makes applicants stand out is clear evidence that they know how to apply such techniques within our particular industry to answer business-relevant questions. This is also a case where the advice I just gave about trying your hand at making something stands - if I see two resumes for an internship on my team with equal education, experience, and technical skills but one of them won an award for a short film they made at a small film festival, which one do you think I’ll be more interested in talking to? Exactly.

Next, never be afraid to think weird. I half-jokingly like to say my role model is Doctor Emmett Brown or “Doc” from Back to the Future. People are often so afraid to sound stupid that they mentally box themselves into thinking in more constrained and conventional ways than they need to. I got over this fear a long time ago, and at this point I’m confident one of my selling points is the fact that there will be truly insightful gems worth following up on consistently scattered across the array of out-there ideas I throw out there. Don’t hesitate to look for and feel out parallels between domains, where an approach from one domain could be translated over to another, and voice such ideas to those around you.

In a similar vein, remember that fortune favors the bold. For better or for worse, achieving our goals almost always involves some small element of luck. But in my opinion, the more you try, the more you reach out, the more energy you put out into the ether to achieve your goals in conventional and unconventional ways, the more likely it is that lady luck will smile upon you to help you get you to your goals. I look back now and count my blessings realizing that the odds of ending up back in the city I wanted to be in, with a job I’m good at, working in the industry that I wanted to be in, making enough to be comfortable immediately after my PhD were astronomical. This advice also intersects with what I said before about having a long-term view - if you plan in advance, move boldly, and persist resiliently, as cliche as it may sound, you’d be surprised what you can manifest into reality.

Lastly, follow your heart and pursue your passions. Take a moment to read this quote from Ralph Waldo Emerson:

The voyage of the best ship is a zigzag line of a hundred tacks. See the line from a sufficient distance, and it straightens itself to the average tendency. Your genuine action will explain itself, and will explain your other genuine actions. Your conformity explains nothing. Act singly, and what you have already done singly will justify you now.

I went from wanting to write music for movies in high school, to attending film school in undergrad, to media research and statistics in grad school, to entertainment data science in the workforce. Each passion I pursued seemed to organically snowball with everything I’d done before to lead naturally to the next thing. Those pursuits of my various passions were the zigs and zags of my life, and I’d like to think overtime they’ve straightened out to a solid average trajectory. I hope such a relentless pursuit of your passions brings you success in your life as well.

Enjoyed this post but even a free subscription feels like too much commitment? It’s okay, we can keep this a situationship for now, but it’d be great if you could:

Buy Me A Drink! 🥂

Otherwise, subscribe for free (or paid!):

I’m actually a bit of an AI skeptic, or at the very least more conservative in my fandom of the technology. By virtue of my job and background, I’m fortunate to have (slightly) greater insight on the inner workings of AI than do many, and maybe it’s my inner reductivist, but it’s hard for me to not think of them as glorified Next Word Predictors in many ways (because they kinda are!). What’s funny is that on the other hand, despite my generally reserved enthusiasm about the technology, I also strongly believe that they’re capable of capturing, reflecting, and giving us insight into latent social, cultural, psychological, and behavioral phenomena and dynamics in previously unfathomable ways (e.g. see research on synthetic data and simulated social environments).

A year or two ago, I started being annoyed with Google and with myself that I had missed the window in which to opt-in to keep legacy free access to G Suite as integrated with my personal website. As a result, I’d started getting charged a few bucks a month for Google Workspace Business Starter, a cost I was resigned to paying as my research showed just how hairy it would be to migrate everything off it to something else. There are few things in the world I hate more than forced subscriptions.

At first, there weren’t really any features in the paid version of Workspace that stuck out to me, features that would make me go out of my way to integrate them into my workflow. The 30GB storage was nice and all, but I was already too committed to Dropbox with my documents with little desire to move everything over. So it seemed that I was paying a monthly fee just to not have to move all my stuff somewhere else.

Then I remembered Gemini. I’d used it a little for work before, but then I realized my Workspace account gave me access to a slightly nicer version of Gemini. Most importantly, it was a version of Gemini where my inputs would not be used to train future Gemini models. This meant that I was free to bounce ideas off it without that paranoid worry that it might get jacked by the good ol’ boys at Mountain View (unlikely, I know).

Since then, I’ve actively started taking steps to integrate Gemini more tightly into my day-to-day and work processes. I even installed the Gemini app on my iPhone, which legitimately is the first AI chatbot I’ve installed on my phone (no, I never installed ChatGPT). I’ve shifted all my personal AI use to Gemini, though I maintain ChatGPT use for work as we have enterprise accounts on the service. Between those two services, I’ve developed several consistent use cases for AI that I find save me a lot of time and frustration.

Writing Research Assistant

I like to write for fun… in case you couldn’t tell. Ahem. Not just non-fiction as you see me post on here, but I also really like to write fiction of various kinds. I will never, ever use AI to write entire pieces for me - and for the record, I was shitting on Grammarly ads long before AI was popular - but I have come to really appreciate AI’s ability to help you get certain details right.

What do I mean by that? I’m not a published or famous writer by any means yet (said every writer ever), but when I write something set in a particular place or time, I like to put in the work to make the background details (e.g. technology, fashion, culture, etc.) sound plausible. Similarly, if I have to make up the details of some fictional technology, I want to make sure the broad details sound realistic and that technical jargon is used in a reasonably grounded manner. All the hundreds of gigabytes of training data that AI models have gone through make them really useful for this purpose, basically giving you a pseudo-expert in a range of fields to sanity check your conjurings. This writing companion perk ultimate isn’t solely limited to fiction. It can also help you pick out, for example, a good tagline for your website, or help you figure out a voicing for your bio that sounds true to you.

Code Accelerator

I code for a living, but I’m most definitely not some master coding whiz. I probably Stack Overflow and Google as much as every other professional coder does and in reality get by perfectly fine in the day-to-day, but that central anxiety of, “Oh my god what if I finally run into some issue I can’t figure out or debug and I look like a dumbass and lose my job?” never fully went away. Even when I do figure things out, there are many who can move far faster than me.

Enter AI. I never vibe code, nor do I ever ask AI to do work in a language I don’t personally know, but it is still a game-changer. Debugging assistance is a basic application, but what I love so much about it is that it can basically spit out already setup boilerplate code for functions that you can directly adapt into your code. No need to waste time piecing together code chunks from different pieces of documentation just to get some basic functionality up and running. You just tell it what you want the code to do, within what parameters, and even with what names if you already have some lead-in code setup, then ba-da-boom, it’ll generate an often instantly functional chunk of code for you.

(Statistical) Concept Explainer

Given what I do for a living, people often laugh when I say that coding and math are… not my favorite things. My anxieties about the former I mention above, and my distaste for the latter? Well, I hate the image of singular objectivity and correctness the discipline cultivates in you in its early phases, just to pull out the rug from under you like, “Jk, there can be multiple, infinite, or no correct answers!” once you get really deep into it. If I wanted to study something where there could be multiple, infinite, or no correct answers, I would’ve studied art instead!

I can make my way through machine learning and statistics papers that heavily draw on math if I really need to - you’d be surprised how many of those fancy equations just tell you to do the same thing repeatedly for different combinations of things - but it’s really not one of my favorite things to do. Now, I can not only get AI to explain an ML/stats concept step by step and ask it questions as we go, but it can also give me examples of applications or connect the concept to particular domains of interest. Even better, relating to the previous section, AI can help me figure out how to best implement certain ML/stats techniques in code!

Trip Planning Advisor

Travel planning is how I first got the inspiration to start this Substack (see article linked below), and it’s another context in which AI can be incredibly helpful. In pre-AI times (wow, that makes me sound ancient), I would put my expertly-honed PhD-level Google-fu skills to work for such purposes. Ultimately, such processes boiled down to middle-clicking top Google search results into a bajillion different new tabs then looking for overlapping recommendations across multiple sources.

But things get trickier when you’re trying to figure out paths less traveled. Landmarks in London? Metro lines in Paris? Easy enough. But if you wanna figure out that bus to Liechtenstein? The best karaoke bar in Krakow? The best path from Tallinn to Riga? Topics like that can involve haphazardly clicking on search results or piecing together tidbits from barely attended forum threads. I’ve found that AI is helpful in at least pointing you in the right direction to look when to comes to such more niche travel topics, giving you at least an idea of what venues to look up, what neighborhoods to consider, and what routes could be an option.

Hbu?

All that is just me getting started. I’m sure I’m gonna come up with more clever uses for AI as I keep using it, maybe setup some custom Gems I go back to for certain routine tasks. And of course, all that is speaking strictly in the Gemini context - there are all sorts of other AI tools, with new ones popping up practically every day. I also look forward to properly experimenting with video, image, and audio generating AI tools at some point.

If you have any novel AI use cases you think I’d find useful, feel free to share them in the comments!

Enjoyed this post but a paid subscription is too much of a commitment? Grab me a coffee (or a drink, if you insist🍸) here!