Pragmatic & predictable notebook using Python environments

Going through the second half of the course of fastai, I was running into problems running the notebooks. I am relatively new to Python package management, and so some of the very cryptic error messages baffled me on how to resolve. 

An example

When trying to run the notebook #9 (may have failed earlier) from part 2, I got errors around NoneType object not being iterable when getting the data back from the dataloader. I was trying to ensure that I had the correct versions of libraries installed, so I went to the terminal window and tried a pip3 install of pytorch libraries, but then that would give this error:

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a Python library that isn't in Homebrew,
    use a virtual environment:
    
    python3 -m venv path/to/venv
    source path/to/venv/bin/activate
    python3 -m pip install xyz

First, I realized after much searching on the internet that the above error message is a passive way (if you wish to install a library) of indicating that you should set up a separate Python environment where you can have specific libraries installed.

Second, I have installed some newer dependencies than what the fastai course expected (torch version needs to be 2.0.1 and numpy needs to be less than 2). So, I needed to have those legacy versions installed to run the fast.ai code. (example python3.12 is installed on my machine which is incompatible with torch 2.0.1).

So, I was wrestling with how I can specify those specific versions in the Jupyter runtime. Python environments help resolve this issue. The premise of Python environments is that you can independently manage all of the library versions for a particular environment without destabilizing other Python libraries you may need access to. You can think of it as a container which allows you to install specific versions that are only available within that container.

However, I had never worked with Python environments before, so I had to learn that. They are pretty easy to do (step 2 is an example below). I was able to isolate all of the package versions using the Python environment so that I could independently run the notebooks without additional problems. The steps below allowed me to do that.

figure 1: how to Python env allows you to target specific libraries

Solution

Steps to get a running local notebook (Macbook) for fastai/course22p2 in the Github repo:

  1. Get Python 3.10
    • brew install python@3.10
  2. Create a virtual environment which is python3.10 based
    • In the terminal window run
      • python3.10 -m venv .fastai2
      • source ~/.fastai2/bin/activate
    • Note: this creates a virtual environment from which you can install pre-reqs which could be incompatible with the currently installed default python version. For example, pytorch 2.0.1 is incompatible with python3.12.
  3. Install the pre-reqs (in terminal using Python env)
    • pip3 install -Uqq git+https://github.com/fastai/course22p2
      1. (Installs all of the miniai components created in earlier notebooks)
    • pip3 install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 
    • pip3 install matplotlib
    • pip3 install jupyter
    • pip3 install numpy==1.23
  4. Run jupyter (in terminal using Python env)
  5. Launch colab in browser
  6. Run the cells in the notebook, voila!

I hope this helps someone save time from figuring out these dependencies in the future.

Prioritize or Perish: The Ultimate Guide to Time Management

The greatest superpower anyone has in this day and age is disciplining their attention and use of time. Time management is the skill that differentiates people who achieve their goals from those who don’t. The key skills to learn for great time management are identifying, prioritizing and executing tasks. Since this particular post is targeted towards employees newly entering the job market, I will focus on the last two prioritizing and executing. Once you have mastered the prioritizing and executing skills, moving on to identifying and exploring new potential priorities will make you unstoppable. The process of identifying tasks will be covered in a subsequent post. 

First, we’ll start with the Eisenhower matrix. This matrix is a great tool to help you understand and prioritize what to work on. 

   Urgent   Not Urgent 
  Important   1 – do   2 – schedule 
  Not important   3 – delegate   4 – delete 
Figure 1: Eisenhower Matrix

Every task should fit into one of the four quadrants:

  1. Is it important and urgent? Do it.
  2. Is it important, but not urgent? Schedule time to complete it.
  3. Is it not important, but urgent? Delegate it.
  4. Is it not urgent and not important? Delete it (don’t do it)

Some excellent books which gave some great nuggets on time management: 

  1. Seven Habits of Highly Effective people
    • Habit #3: Putting first things first means organizing and executing around your most important priorities. 
  2. Getting Stuff Done
    • Write a list of specific things you want to accomplish to get it out of your head. E.g. Write next blogpost draft on time management skill. 
  3. Checklist Manifesto
    • Checklists provide discipline, and discipline makes success possible. 
  4. Deep work
    • “The key to developing a deep work habit is to move beyond good intentions and add routines and rituals to your working life designed to minimize the amount of your limited willpower necessary to transition into and maintain a state of unbroken concentration” 

Each of these books circle around a theme of discipline around writing down and prioritizing what you should be focused on. I have synthesized the advice into three key skills to master. 

These key skills are: 

  1. writing everything down which you need to do (I.e. organizing or identifying tasks) 
  2. prioritizing tasks  
  3. executing tasks in prioritized order 

Make a list, prioritize, and execute. Repeat. 

1) Writing everything down  

If you aim at nothing, you will hit it every time.

Zig Ziglar

Let’s take an example, first write down everything that you need to prioritize over the next week. Here’s an example list of my tasks: 

0. Finish draft of time management with focus on prioritizing and executing 

  1. Drop off Leaf for repairs at auto-body shop (Mon) 
  2. Drop off Dad @ airport (Mon) 
  3. Clean-up
    • Clean desk 
    • Vacuum desk area 
  4. Training
  5. Network
    • Go through list of previous contacts (Google sheet) and schedule three informational interviews (follow-up on potential introductions) 
  6. Prepare for interviews (obfuscated company names to variables to avoid exposure)
    • CompanyX – interview Tuesday 
    • CompanyY – interview Thursday  
    • CompanyZ – interview Friday
      1. Each of these contain subtasks around looking up people I’m meeting with exploring their business, preparing questions for the interviews, etc. 
  7. Call Rightway pharmacy (re: compounding prescription re-imbursement) 
  8. Check stock market once this week 
  9. Read through Twitter feed for 30 mins max 
  10. Engage with a couple of posts on LinkedIn for 30 mins max 
  11. < Personal family / friend related priorities > 

The most important step in my week is cataloging what things need to be done. When I was younger, and had less responsibilities, it was easier to juggle all of these tasks mentally. However, as I grew in responsibilities (more kids, employees, job responsibilities) there was an inflection point where I realized the lack of an organized list was hurting my impact. I would focus on whatever I wanted to at the moment, and sometimes that was the wrong thing. Focusing on the wrong thing is easier to do when you don’t have a list. Alternatively, I would prioritize interrupt driven requests without weighing it against my other weekly priorities. At the end of those weeks, I would look back retrospectively and wonder where all my time and energy had been spent, and what impact I had to show for it.  

2) Prioritizing tasks 

The second most important step when making the list is to prioritize it according to the Eisenhower matrix. Here’s an example of bucketizing my items:  

  • Urgent & important – items 0, 1, 2, 6 
  • Not urgent & important – items 4 & 5 
  • Not important & urgent – item 3 & 7 
  • Not important & not urgent – item 8-10 

Note: I added items 8-10 as an example of areas where I personally struggle with setting boundaries.  

Note 2: item 11 is expanded on the longer list, and this particular week had many additional family responsibilities that aren’t normally present. 

Of course, just because you can identify some items as delegate in the matrix, doesn’t mean you have the means or have the time or emotional overhead of delegating the task. For example, task 3 (clean-up) although it’s categorized as “not important” and “urgent”, my wife and kids wouldn’t necessarily take kindly to me asking them to clean up the mess I made at the desk. So, what do I do with those tasks? I call these filler tasks. I fold these filler tasks into my week when I’m feeling less motivated to do a more thought or focus intensive task. Completing some of these tasks and checking them off can significantly boost my motivation and help defeat procrastination.  

Having a deadline associated with tasks, will give you the proper focus to ensure that you avoid doing non-important tasks. For example, I know my dad’s flight leaves on Monday morning, so I schedule time for “drive dad to the airport” on Monday morning in my calendar. I’m sure that my dad appreciates that I focused on dropping him off at the airport on Monday, rather than waiting to bring him to the airport on Tuesday. Additionally, having this context will help you to order similar tasks. For example, the interviews with the companies (X, Y, and Z) were actually on Tuesday, Thursday, and Friday, so I needed to focus on preparing for X first, since it was on Tuesday. Getting that order incorrect would have left me unprepared for the interview. 

People think focusing is about saying yes to the thing you’ve got to focus on. But that’s not what it means at all. It means saying no to the hundred other good ideas that there are. You have to pick carefully. I’m actually as proud of the things we haven’t done as the things I have done. Innovation is saying no to 1,000 things.

Steve Jobs

3) Executing tasks in prioritized order 

It can be helpful to have a strict ordering of the tasks on your list.  This will allow you to focus on only the next highest priority item on the list and avoid getting to the end of the week without a critically important task done. I am less rigid about this strict ordering, just because I know how I work best. I like autonomy to choose what I’m working on next instead of having a schedule dictate what that will be. Some days I will wake up and I have to coax myself more into getting into the zone of work. This may mean that I focus on getting a couple of the filler tasks done as a warm up to the more difficult tasks. This is the exact opposite advice which Cal Newport gives in Deep Work, where he suggests putting the most difficult task first in the day. However, I know what works well for me, and sometimes that is a gradual easing into the Deep Work.  

It’s important to focus on the priorities of the tasks when choosing what to work on next. 

Allowing less (or not) important (urgent or not urgent) items to choke out your ability to focus on important items will stifle impact. Distractions impact everyone in this era where every app and device is vying for our attention. Some of the most successful people that I know are ruthless in training their focus on executing tasks in the important row (see Eisenhower matrix). A negative example, a newer employee Bob would spend a lot of his time on his cellphone texting with friends during the day. While this isn’t a problem for some people, sending a one-off text, or doing it during lunch time. Bob would become absorbed in these conversations and spend hours on his phone, texting with friends, and it was impacting his ability to prioritize and get his work done. I noticed this lack of progress when he shared during daily stand-up. In private I asked him to measure where he was spending his time. This insight led to an important break-through (I’m spending too much time on not important / not urgent tasks, i.e. texting friends) for the person in time management and impact. *  

* Note: – An insight I learned when doing performance optimization in software, you have to measure where the time is being spent with a profiler, I.e. you can’t guess. You can try to guess where the software is spending time, but you will often be wrong. As an example, at Microsoft, I found logging boilerplate in a large service codebase which I worked on was doing stack trace calls to determine in which file it was making the request to log the filename. The stack trace call in C# is expensive because it uses reflection, so I just added the file name logic to be statically built. I would never have guessed that it was the stack trace call that was eating up so much of the compute without measuring it! 

Today, there are many ways to observe what you’re working on. The simplest is setting a fifteen minute timer which goes off throughout your day. When the timer goes off, immediately write down what you’re doing (even if it’s not work related, e.g. searching for a product on Amazon), and then go back to whatever you were doing. After a few days of observation, you might observe a pattern, like Bob, who discovered he was spending a LOT of time on his phone. (There is now tooling support for this measurement: Toggl

There are various reasons (ignorance, lack of discerning priorities and impact of tasks, being interrupt driven) for folks having trouble with time management. However, knowing what to do, when it should be done, and in what order they should be done will help provide the structure and the motivation to say no (especially to things outside of the urgent / important bucket). Giving yourself the permission to say “no”, to good, but not the best things will help you to be more effective and intentional in your impact. 

FAQs:  

What happens when a new request comes in from your boss? 

Part of the power of having a prioritized list that you are working towards is allowing that list to help you say no. After putting the task through the Eisenhower matrix, you can take the new task and measure to see where it compares to your other items. If it’s being asked of you by a manager, it might be worthwhile to have a trade-off discussion with them about what you planned to work on that might be impacted by the new request. If it doesn’t make the list, consider adding it to your next week’s priority list, or better yet seeing if it can be deleted. 

What about recurring items? 

I try to put all recurring items on my calendar. This avoids me having to think about them, and gives me greater confidence when scheduling meetings, etc. If I have to take my daughter to/from soccer practice, I put those commitments as a recurring meeting into the calendar to help me remember to do them, and then also keep the list to a manageable length. 

How do I improve my discipline in creating, prioritizing and executing against a list? 

Start off slowly, you don’t have to do it perfectly and capture all the context the first time. Create a list with a few items, prioritize and execute. Repeat this regularly over the next two months. Revisit what is working well versus what is not.