61: Perfectly Neutral

18 Apr 2014

Vinyl sound, Apple/Samsung trial documents, Greg Christie’s departure, pCell, performance vs. scaling, and why Marco never uses JOINs.

Episode Description:

• Follow-up on Photo Stream Web Sites
• Follow-up on vinyl, including how Marc Edwards thinks it's evil, and the difference between pleasing and accurate sound. (Marco's open and closed headphones, amp, and DAC)
• The Apple insight attained by way of documents released in the Samsung trial
  • Creativity, Inc., the book about Pixar by Ed Catmull
• Is Greg Christie's departure a contentious story or not?
  • This, too, relates to Pixar
  • Ken Ferry on getting ideas approved at Apple, as heard on Debug #33
• Artemis pCell (demo video)
• Scaling servers and how to use MySQL effectively
  • Marco's preferred book (but he prefers the second edition)
  • Perhaps not use MySQL?
  • A long list of MySQL gotchas
  • How FriendFeed did crazy schema-less MySQL
• After show: why Casey hates his car, thanks to a trip to the movies.

Sponsored by:

• PDFpen for iPad: Powerful, mobile PDF editing.
• Igloo: Igloo is an intranet you'll actually like.
• New Relic: See your web app's performance and find bottlenecks anywhere in the stack. Start your 30-day free trial.

Transcribed using Whisper large_v2 (transcription) + WAV2VEC2_ASR_LARGE_LV60K_960H (alignment) + Pyannote (speaker diaritization).

Transcript start

⏹️ ▶️ ^John So your tweet about us having thin topics like oh, there’s a lot of stuff We didn’t even get the Apple wearables, but after that

⏹️ ▶️ ^John you’re right then we’re out of topics. I Just have one item I guess it

⏹️ ▶️ ^John was a popular item, but just one last week. We were talking about How if you

⏹️ ▶️ ^John have some sort of service that? Deals with your photos and lets you share them no matter what cool features you have

⏹️ ▶️ ^John you got to at least Also have a way to send a link to somebody they can view in any web browser

⏹️ ▶️ ^John And we also talked about Photo stream and iOS integration and stuff like that

⏹️ ▶️ ^John And although we didn’t come out and say it directly the implication was that photo stream did not have a public URL But

⏹️ ▶️ ^John photo stream does have public URLs You can make a little URL for our website that you can share with somebody

⏹️ ▶️ ^John that lets them see the pictures in your photo stream I Use photo stream with my family

⏹️ ▶️ ^John and my family all has iOS devices and the great thing about that is It notifies you on your phone or on your

⏹️ ▶️ ^John iPad and when you get the notification and you can just swipe and see the pictures immediately. It’s a much superior

⏹️ ▶️ ^John interface for viewing pictures, but you’ve always got to have that web interface underneath it all. And this came up

⏹️ ▶️ ^John in the context of Carousel, which I think, Casey, you said they didn’t have a way for you to just get an easy web link?

⏹️ ▶️ ^Casey That was my experience. That’s correct. So I had emailed what was to Carousel

⏹️ ▶️ ^Casey a completely random email address that wasn’t associated with a Dropbox account. The email address,

⏹️ ▶️ ^Casey in fact, was my work email. And when I received the email from personal

⏹️ ▶️ ^Casey me, and I’m doing that in air quotes, to work me, the work email basically said, hey, you need to install Carousel

⏹️ ▶️ ^Casey if you wanna do anything useful. And I think it did have like six or seven pictures in it, little

⏹️ ▶️ ^Casey thumbnails of the pictures that I’d actually selected. So it wasn’t like stock photography or anything like that. But with

⏹️ ▶️ ^Casey that said, it didn’t do anything and there was no web-based link. It basically

⏹️ ▶️ ^Casey said, go to the app or get the hell out of here.

⏹️ ▶️ ^John Does anybody have any more follow-up? Casey, do you feel like you need to follow up at all about

⏹️ ▶️ ^John any topics from last week?

⏹️ ▶️ ^Marco Do you know what you did, Casey? Do you want to tell us all what you did?

⏹️ ▶️ ^Casey I’m sorry, daddy. No, I don’t know. I’m a little –

⏹️ ▶️ ^Casey not bitter. I’m a little frustrated with the feedback from last week and I was

⏹️ ▶️ ^Casey debating talking about it and since you’ve prompted me, I’ll go ahead and do so and just make my

⏹️ ▶️ ^Casey world even worse for myself. And the thing that frustrated me about the

⏹️ ▶️ ^Casey feedback we received is that there were a few instances, and Mark Edwards

⏹️ ▶️ ^Casey is from Bajango, Django, I always pronounce it wrong, I’m so sorry Mark. But anyway, he wrote

⏹️ ▶️ ^Casey a relatively long missive that he put up as a gist, a gist, whatever. What

⏹️ ▶️ ^Casey am I thinking of? Trevor Burrus Just. John Sandusky Thank you. I doubted myself. Anyways, I’m all out

⏹️ ▶️ ^Casey of whack. So he put up a gist and he explained as someone who used to do this sort of thing for a living

⏹️ ▶️ ^Casey why vinyl is basically the most evil thing in the world. And while I

⏹️ ▶️ ^Casey don’t entirely agree with what he said, I can’t really factually debate

⏹️ ▶️ ^Casey it. And so things like that, and we’ll put a link in the show notes, that was actually welcome and

⏹️ ▶️ ^Casey useful. But I got a lot of feedback, we got a lot of feedback, and I got a lot of feedback,

⏹️ ▶️ ^Casey which basically amounted to, I read something on the internet once, it

⏹️ ▶️ ^Casey must be true, you’re crazy and stupid.” And that kind of bothers me a little

⏹️ ▶️ ^Casey bit because it was oftentimes not really based

⏹️ ▶️ ^Casey in fact, or sometimes it was, but never ever ever based on experience. And I’m

⏹️ ▶️ ^Casey not going to try to rehash the argument, but suffice to say I put a lot more stock when I receive

⏹️ ▶️ ^Casey feedback from someone who has had the experience of listening to vinyl on a

⏹️ ▶️ ^Casey really nice setup over someone who has just read some things or seen some stuff on the internet

⏹️ ▶️ ^Casey and swears that that’s true. Now yes, there’s a lot of science behind it. Yes, I understand

⏹️ ▶️ ^Casey that. Yes, I’m probably crazy. All of that is fine. I agree. I understand it.

⏹️ ▶️ ^Casey It was probably silly of me to say that the fidelity of vinyl was better than the fidelity of

⏹️ ▶️ ^Casey CDs. But when it comes down to it, something I said on Twitter, which

⏹️ ▶️ ^Casey I actually was fairly proud of myself, I’m patting myself on the back, is that it’s very kind of cliche,

⏹️ ▶️ ^Casey but life is what happens between one and zero in my eyes. And whether or not the accuracy

⏹️ ▶️ ^Casey of the reproduction on vinyl or CD is better, I just happen to prefer

⏹️ ▶️ ^Casey the feel and to some degree the, what did you call it? The tea something? I

⏹️ ▶️ ^Casey already

⏹️ ▶️ ^{Casey, John} forgot.

⏹️ ▶️ ^John Tea ceremony. Japanese tea ceremony?

⏹️ ▶️ ^Casey Google it. Yeah, no, I know what you mean. So, I prefer the tea ceremony of it. And

⏹️ ▶️ ^Casey the only thing I will say to defend myself other than come at me once you’ve had personal

⏹️ ▶️ ^Casey experience you damn nerds the only other thing I’ll say is

⏹️ ▶️ ^Casey Why is it John that you choose to drive a manual transmission in 2014?

⏹️ ▶️ ^Casey That’s

⏹️ ▶️ ^Marco not even close to a fair comparison

⏹️ ▶️ ^John I Was gonna let you slide but you come you come back at me with this and you just like What I was gonna

⏹️ ▶️ ^John let slide is the terror that thing with life if this is what happens between one and zero, that like, that vaguely

⏹️ ▶️ ^John alludes to terrible, illogical reasoning behind crazy,

⏹️ ▶️ ^John you know what I mean? Like, I know that’s not what you’re getting at at all, like I understand, but the fact that you chose to use one

⏹️ ▶️ ^John and zero is like another nudge at like the, come on guys, we all know that digital stuff is just cold

⏹️ ▶️ ^{John, Marco} and unfeeling

⏹️ ▶️ ^John and

⏹️ ▶️ ^{John, Marco} hell, you know what I mean? Like, there’s an edge to

⏹️ ▶️ ^John that. But then you came at the other thing, like I found the feedback frustrating as well, and you would think I wouldn’t find the feedback frustrating

⏹️ ▶️ ^John because I think that most of the feedback agreed with Marco and I, but I found that feedback very frustrating because

⏹️ ▶️ ^John I mean, this is a common phenomenon feedback for the show. Very often we will get feedback no matter what the topic

⏹️ ▶️ ^John on past shows and everything where someone will be. I mean, it’s not bad feedback. It’s good because it shows enthusiasm.

⏹️ ▶️ ^John You know, someone who’s a big fan of the show will write in and in some ways this is the best

⏹️ ▶️ ^John kind of feedback that will be like listening to your show. Maybe think about this and here’s what I think about it and they’ll

⏹️ ▶️ ^John present a bunch of ideas about the topic. But what I find frustrating is a lot of the time,

⏹️ ▶️ ^John the ideas that they present are things that we talked about on the show. And you don’t want to write back

⏹️ ▶️ ^John and say, yeah, we talked about that, this exact thing, at length on the show.

⏹️ ▶️ ^John But then it sounds like you’re not valuing their feedback, and it’s good that they’re enthusiastic about it. But sometimes it’s frustrating to see, like, I

⏹️ ▶️ ^John obviously failed to communicate this idea, because it keeps coming back to me as if it’s information that we are

⏹️ ▶️ ^John missing, or if it’s new information or new insights. And it’s like, we talked about it for 15 minutes. We

⏹️ ▶️ ^John said almost these exact words. And it’s like, sometimes it’s people sending feedback in real time, where they listen, and then I

⏹️ ▶️ ^John guess they pause it or whatever. And then they write their three-page missive, and then they start playing again.

⏹️ ▶️ ^John And then because they haven’t gotten to the end of the podcast and realize we addressed the point. And that happens with tweets.

⏹️ ▶️ ^John Obviously, that’s fine. Sometimes it happens with emails too. But a lot of the feedback was like

⏹️ ▶️ ^John presenting points back to any of us, me, Marco, you, that I thought I could

⏹️ ▶️ ^John swear we covered on this show. Like, we talked about these exact things. And it’s like, maybe we’re not

⏹️ ▶️ ^John communicating effectively, or just like we’re talking past our audience or something.

⏹️ ▶️ ^John And in the same way, I saw a lot of arguments directed at both Casey and us that I thought we talked about

⏹️ ▶️ ^John on the show and addressed all different sides of coming back at us.

⏹️ ▶️ ^John Arguments that I thought I had refuted and settled down, arguments that Casey never made,

⏹️ ▶️ ^John arguments that we never made, positions that we never staked out, You know, the whole nine yards. This is obviously

⏹️ ▶️ ^John a very fraught issue. And I think, Casey, the problem you’re still having with it, and the problem

⏹️ ▶️ ^John you’ll continue to have with it, is that you just got to like, you have to just make it clear what it

⏹️ ▶️ ^John is. Like, you know, it’s like in the world of internet, everybody’s like moving the goalposts. You got to make it clear what everybody’s talking

⏹️ ▶️ ^John about. Because at this point, I think everybody on this podcast all agrees. It’s just that you keep wanting to put the goalposts over there

⏹️ ▶️ ^John and pretend that’s what we were talking about. And me and Mark will keep wanting to put the goalposts over here and pretend this is what we’re talking about.

⏹️ ▶️ ^John And I’m trying to keep them kind of where the original thing was, which was the whole idea on our L talk, where

⏹️ ▶️ ^John Faith said, I’m not going to buy a device that purports to have high quality music. And you’re right, fidelity

⏹️ ▶️ ^John is a better word. It was pointed out to us by Dr. Drang. We should have been using that word, which could be part of the communication problem here.

⏹️ ▶️ ^John But anyway, I’m not going to buy a device like the Pono, because if I want high quality, meaning high fidelity

⏹️ ▶️ ^John music, I’ll just listen to vinyl. And that, I think, is a clear implication that we’re talking

⏹️ ▶️ ^John about the Pono, and the whole big deal with that is there’s more information in the music.

⏹️ ▶️ ^John And it’s like, well, if I want more information in the music, if I want higher fidelity, if I want a

⏹️ ▶️ ^John more accurate reproduction of sound, I will go to vinyl. And you seem

⏹️ ▶️ ^John to follow down that trail. And my objection and Marco’s objection is that vinyl is not good at reproducing sound,

⏹️ ▶️ ^John and not as good as CD. And all this other stuff I think are great for us to talk about, and it’s good to define

⏹️ ▶️ ^John the boundaries of what we believe in everything. But you should be happy with that. It’s not as if we like

⏹️ ▶️ ^John it’s it’s not as if you should you shouldn’t cling to that idea like that fidelity thing. If that’s

⏹️ ▶️ ^John not what you’re talking about, then fine, then you agree with us there and we agree with you over the tea ceremony and everybody’s

⏹️ ▶️ ^John happy. And that one of the things that he keeps getting thrown back in is like

⏹️ ▶️ ^John that. I like to like to reframe a lot of these people’s arguments. And the popular one is like if

⏹️ ▶️ ^John I said, look, if I’m really interested in high fidelity images,

⏹️ ▶️ ^John I’ll go to Instagram. Like that’s effectively the argument they’re making. Like, you know,

⏹️ ▶️ ^John I don’t need a full frame camera with a really great sensor and lots of megapixels.

⏹️ ▶️ ^John If I’m interested in photographic fidelity, I will look at a camera from a picture of 1992 on Instagram.

⏹️ ▶️ ^John Like that’s the argument that many people are making, and unknowingly that they’re making, that they’re completely combining

⏹️ ▶️ ^John what they think looks good, which may be like a super or grainy Instagram filter, and the concept of fidelity,

⏹️ ▶️ ^John which is a straightforward thing that can be measured and talked about in objective terms, more or less.

⏹️ ▶️ ^John And people coming back with, like, I love tube amps and everything like that, instead of digital amps, like, that’s not what we’re

⏹️ ▶️ ^John talking about. It’s like, whatever it is that you think sounds good, all we’re talking about is, given that

⏹️ ▶️ ^John sound, and I probably made a mistake using live instruments because I had to deal with all the people talking about live music. Given

⏹️ ▶️ ^John some sound that you think is awesome, what is the best way to transport that

⏹️ ▶️ ^John sound through space and time to reproduce it elsewhere. That’s all we’re talking about. And it has nothing to

⏹️ ▶️ ^John do with whether you like tube amps or solid state bands, whether you like six string guitars or five string, whether you like someone singing through

⏹️ ▶️ ^John a paper bag or whatever the sound is, you’ve produced a sound, a song, live, recorded, whatever. You want

⏹️ ▶️ ^John to get that across space and time to someone else. You want to put it on something so that that sound that you love

⏹️ ▶️ ^John can get to them exactly as it is. Does it have whatever that sound is? Master it however you want, do whatever you want. How do you

⏹️ ▶️ ^John get that sound? CD versus vinyl. That’s what we were talking about. Everything else is like immaterial.

⏹️ ▶️ ^John I don’t care if you like the sound of someone singing through a paper bag. Or

⏹️ ▶️ ^John I care all. No, no, no, I’m talking about vinyl. I don’t care if you like Tom Waits. I don’t care

⏹️ ▶️ ^John what kind of music you want. I’m just saying, you have something there. And again, live music was easy because it’s like, oh, I hear

⏹️ ▶️ ^John that now. You just want to put that on something so it could be played back elsewhere. And if your choices are CD

⏹️ ▶️ ^John and vinyl, and you care about the fidelity of that reproduction, you should pick CD instead of vinyl. And

⏹️ ▶️ ^John that’s it. And everything else people want to talk about, it’s so hard for them to maintain focus. It

⏹️ ▶️ ^John was hard for us to maintain focus, because we have drifting. But it’s a difficult topic to talk about, I think.

⏹️ ▶️ ^Casey And I think you’re right. And I think you’re right in saying that we all agree that if fidelity

⏹️ ▶️ ^Casey is truly the issue, that I can get behind the science, even though I give it

⏹️ ▶️ ^Casey a little bit of side eye. Nevertheless, I can get behind the science that says I’m crazy and that

⏹️ ▶️ ^Casey digital mediums are the better, higher fidelity method of reproducing

⏹️ ▶️ ^Casey music. But just as you said, I actually happen to prefer the tea ceremony, the

⏹️ ▶️ ^Casey emotion behind it. I was talking to my wife, Erin, about it, and she said, well, you know, you have to consider you grew up with vinyl, and this

⏹️ ▶️ ^Casey was something that carries a lot of emotional baggage. I think John, or one of you guys, might have said that last week.

⏹️ ▶️ ^Casey Anyway, it carries a lot of emotional baggage that for most humans, especially in this day and age,

⏹️ ▶️ ^Casey they don’t have that emotional baggage. And I say baggage actually in a good way in this context. I’m sure you probably think

⏹️ ▶️ ^Casey of it as a bad thing. But nevertheless, I don’t know what but one thing I learned from this and I’m really

⏹️ ▶️ ^Casey being serious is that it is very frustrating To say something which

⏹️ ▶️ ^Casey admittedly is a bit contrarian and admittedly I’d kind of picked this fight So admittedly I

⏹️ ▶️ ^Casey kind of made this bed for myself But man, is it frustrating to say something that you

⏹️ ▶️ ^Casey truly believe and then the whole of the internet decides to come out And tell

⏹️ ▶️ ^Casey you how wrong you are in that in and of itself I can deal with that. That’s fine I’ve been wrong plenty of times will be wrong plenty of times again

⏹️ ▶️ ^Casey But it was very frustrating for me to receive a whole bunch of feedback from people

⏹️ ▶️ ^Casey who had perhaps never even heard vinyl before in their lives, yet decided

⏹️ ▶️ ^Casey to take the time out of their day to explain to me how wrong I am. And

⏹️ ▶️ ^Casey I actually want – I’m twisting this, not – I don’t want that to sound like a complaint. What I

⏹️ ▶️ ^Casey want it to sound like is I learned a little bit that I think I do that to people

⏹️ ▶️ ^Casey sometimes. And I can’t think of a great example other than maybe snickering at somebody pulling out an Android phone, for example.

⏹️ ▶️ ^Casey But –

⏹️ ▶️ ^Marco Let me get that for you.

⏹️ ▶️ ^Casey Yeah, well, I don’t want to be that guy, and I think I am that guy. And so it was a very good

⏹️ ▶️ ^Casey learning experience for me that I need to stop being that guy. Does that make sense?

⏹️ ▶️ ^Marco Yeah, I think – I don’t know, I mean, you know, we’re always going to get – oh God, I have so much more

⏹️ ▶️ ^Marco to say about this, but I’m trying to resist. We’re always going to get,

⏹️ ▶️ ^Marco you know, people who disagree with what we say. always do every week. I mean, we get, you know, to give the listener some idea,

⏹️ ▶️ ^Marco we get probably, I would say, 20 feedback emails a day these days.

⏹️ ▶️ ^Marco It’s a lot. I’m shocked how much feedback we get from this show. Matthew

⏹️ ▶️ ^Casey And I should very quickly interject and say, I can’t speak for you two, but I try very hard

⏹️ ▶️ ^Casey to read every single piece and at worst, skim every single piece of feedback that we

⏹️ ▶️ ^Casey get.

⏹️ ▶️ ^Marco Yeah, me too. And it really does. When you only get a couple a week,

⏹️ ▶️ ^Marco that’s easy to keep up with. Now it’s getting hard to keep up with. We get so much feedback

⏹️ ▶️ ^Marco now that it’s kind of difficult. Some of them are a few lines and that’s great. Some

⏹️ ▶️ ^Marco of them are like six paragraphs, six dense paragraphs at that. It

⏹️ ▶️ ^Marco takes some serious time to get through some of these. But anyway, I

⏹️ ▶️ ^Marco think we are expressing our opinions in public. Lots of what we say

⏹️ ▶️ ^Marco is going to be argued with. That’s just the nature of putting yourself out there.

⏹️ ▶️ ^Marco I’ve been putting myself out there and my often rash

⏹️ ▶️ ^Marco or incorrect or badly stated opinions out there online for

⏹️ ▶️ ^Marco the better part of a decade now, pretty consistently. And whenever I say something

⏹️ ▶️ ^Marco that everyone jumps down my throat for, for being totally wrong or off-base or

⏹️ ▶️ ^Marco misinformed or just not very well stated, I learn something

⏹️ ▶️ ^Marco from that. And I appreciate that because most people don’t expose themselves

⏹️ ▶️ ^Marco to that much criticism. Most people don’t have an audience to tell them

⏹️ ▶️ ^Marco when they’re wrong and who actually will tell them when they’re wrong, and an audience that’s big and diverse and

⏹️ ▶️ ^Marco smart enough to notice when they’re wrong and to be able to express that in any kind of useful way.

⏹️ ▶️ ^Marco So most people tend to be terrible arguers, have narrow worldviews, etc., etc.,

⏹️ ▶️ ^Marco and it’s hard to get people to change their mind

⏹️ ▶️ ^Marco when they’ve spent their whole life thinking that they’re right and no one has really ever challenged them on that. So

⏹️ ▶️ ^Marco I’m very thankful that we have the opportunity, which most people don’t have. We as

⏹️ ▶️ ^Marco podcasters and anyone else who’s a podcaster or a blogger or any other way that you can put yourself out there.

⏹️ ▶️ ^Marco We are lucky that we have the opportunity to be criticized by a massive

⏹️ ▶️ ^Marco worldwide audience. Even if your audience is 25 people, that’s

⏹️ ▶️ ^Marco still 25 random strangers who don’t care as much about how you feel as your family does and who are willing to tell you

⏹️ ▶️ ^Marco when you screw something up. And you know as the audience gets larger, it

⏹️ ▶️ ^Marco hurts more when you screw up because you have more people telling you that you’re an idiot. But I

⏹️ ▶️ ^Marco think it is useful and it’s useful and so like in your case you know as john said like in your

⏹️ ▶️ ^Marco case you have you know with the vinyl thing like you expressed valid

⏹️ ▶️ ^Marco opinions there however you worded them in a way and you made a generalization

⏹️ ▶️ ^Marco that you couldn’t support and so the feedback kept beating that into your head

⏹️ ▶️ ^Marco basically and it’s true and and it’s you know it certainly was not

⏹️ ▶️ ^Marco certainly was not probably easy to read everyone saying that you were wrong although by the way there are a lot of who agree

⏹️ ▶️ ^Marco with you, I should point out. But, like,

⏹️ ▶️ ^Marco your main point was that it sounds better because it sounded better to you. A

⏹️ ▶️ ^Marco solid argument would have been, you like the sound better.

⏹️ ▶️ ^Casey MW Right. And I think I fell on my face because I painted it, like John was saying and like

⏹️ ▶️ ^Casey you’re saying now, I painted it as a universal statement when I really shouldn’t have, that it was

⏹️ ▶️ ^Casey factually better that it it was superior to CDs and while I might

⏹️ ▶️ ^Casey prefer the Experience and the in the quirks of the sound I don’t think that I should have gone

⏹️ ▶️ ^Casey as bold as I did

⏹️ ▶️ ^John and as one of the feedback things actually a couple people did this like Presented if if we could just take

⏹️ ▶️ ^John that sound that you love and record it onto CD and play it back to you And be in a way that you didn’t know

⏹️ ▶️ ^John whether we were playing back the vinyl or not you would not be able to tell. Like, we would be exactly

⏹️ ▶️ ^John accurate that you’re producing that vinyl sound that you love with every nuance that’s available, you know what I mean? Like,

⏹️ ▶️ ^John and that gets to the heart of the matter. And it’s like, we’re all for you liking whatever sound you like, but

⏹️ ▶️ ^John like, when we’re talking about mediums, when we just say the word vinyl, you’re not talking about a song

⏹️ ▶️ ^John or a sound, you’re talking about a medium, and that’s a different topic. There’s so many different ways to try to come at this,

⏹️ ▶️ ^John to try to get people to understand it, let us keep going back to whichever side they’re on, whether like, no way man vinyl

⏹️ ▶️ ^John is awesome or like vinyl is always evil like people just don’t want to define

⏹️ ▶️ ^John the boundaries of what it is that they’re saying and it’s impossible to come to any sort of agreement or disagreement

⏹️ ▶️ ^John if if no one is defining the boundaries of what they’re saying a lot of like especially the tweet length ones which you can’t blame

⏹️ ▶️ ^John it’s hard to hard talk about this in a single tweet fell victim to that both you know

⏹️ ▶️ ^John on all sides of the debate or the best ones where they say I like tweets on any topic with let’s I’ll say,

⏹️ ▶️ ^John I agree with Casey about this. I agree with Marco about this. I disagree with John about this. And they

⏹️ ▶️ ^John will ascribe to all three of us opinions like that 50% of the time we didn’t take. And it’s like,

⏹️ ▶️ ^John so I agree with Marco that Starbucks coffee is great. I agree with Casey that all cars should

⏹️ ▶️ ^John be black. And I agree with John that Pearl’s a terrible language. That’s a silly example. But it’s like,

⏹️ ▶️ ^John you put our names in a tweet, and you’ve expressed your own opinions. But you’ve, by implication, ascribed

⏹️ ▶️ ^John the opposite opinions to us. and I don’t think we ever, anyway. It’s the excitement of feedback. I

⏹️ ▶️ ^John agree with Marco, getting this feedback, as I’ve said many times, is very valuable. We’re lucky

⏹️ ▶️ ^John to get a lot of it. You too can get a lot of it as well if you just go to online communities and whatever topic

⏹️ ▶️ ^John you’re interested in and just start, like I said, a Usenet in my past, where it wasn’t as big an audience

⏹️ ▶️ ^John as we have here, but it only takes like four or five mean people to put you in your place for a couple

⏹️ ▶️ ^John of years and you’ll get better.

⏹️ ▶️ ^Casey Great. And again, I don’t mean to complain necessarily about getting feedback. I don’t mean

⏹️ ▶️ ^Casey to complain about all of the internet or most of the internet thinking I’m wrong.

⏹️ ▶️ ^Casey That actually doesn’t bother me. What was frustrating was people who really didn’t have,

⏹️ ▶️ ^Casey to be blunt, anything that constructive to add, just coming out of the woodwork and saying, eh, you’re

⏹️ ▶️ ^Casey wrong. And you’re wrong because I read something once. That’s just not helpful. Whereas something that

⏹️ ▶️ ^Casey Mark did, which he said, from personal experience. Let me take you through

⏹️ ▶️ ^Casey why it is you’re wrong. That is infinitely valuable and extraordinarily valuable. And

⏹️ ▶️ ^Casey I’m very thankful for that.

⏹️ ▶️ ^John But the other people are expressing their enthusiasm, though, like even if they’re they’re they’re doing it in a way that makes you feel

⏹️ ▶️ ^John bad. They are they’re saying, A, we listen to the show and B, we’re enthusiastic enough about it

⏹️ ▶️ ^John to seek you out on Twitter and send you something. So there are there is even good in that, I think.

⏹️ ▶️ ^Casey Yeah, I agree. And that’s why even on the emails that I want to reach to the computer and backhand whoever

⏹️ ▶️ ^Casey wrote them. Even on those emails, I try always to force myself to

⏹️ ▶️ ^Casey end the email in such a way that when I write, thank you for listening and thanks for your feedback

⏹️ ▶️ ^Casey that it’s taken as genuine because I do mean it genuinely, even if I want to freaking kill whoever

⏹️ ▶️ ^Casey it is I just replied to. So I don’t know, maybe this is too inside baseball, this might be boring, maybe

⏹️ ▶️ ^Casey we’ll cut it, I don’t know. But it just, the point I was trying to get to was that

⏹️ ▶️ ^Casey it made me look at myself with a bit of a critical eye, which again is

⏹️ ▶️ ^Casey even more reason why even the feedback that I’m bemoaning is actually valuable in

⏹️ ▶️ ^Casey its own way.

⏹️ ▶️ ^Marco And to point out also, I’ve been in this

⏹️ ▶️ ^Marco high-end audio world for a lot recently, and I found that…

⏹️ ▶️ ^Marco reason why I don’t like vinyl and a lot of analogs and analogies, same reason

⏹️ ▶️ ^Marco why I briefly had a chance to have a tube amp in my house for a couple of weeks to

⏹️ ▶️ ^Marco try out and I didn’t like it and ended up not buying one, the

⏹️ ▶️ ^Marco reason why people like vinyl sound, not everyone, but the reason why the people who do like

⏹️ ▶️ ^Marco it like it. In addition to all the things we talked about last week about the tea ceremony and

⏹️ ▶️ ^Marco the romanticism of it and how hip and awesome it looks and everything is slow and artisanal and

⏹️ ▶️ ^Marco manual. In addition to all that crap it does sound different

⏹️ ▶️ ^Marco and it sounds different in various ways that people find pleasing.

⏹️ ▶️ ^Marco Now as but you know that wasn’t the argument that John was making against you however that

⏹️ ▶️ ^Marco was what most people thought when you know with for the argument about vinyl sounding better than CD

⏹️ ▶️ ^Marco or whatever you know in my own purchases and what I found

⏹️ ▶️ ^Marco in my experiences with high-end audio stuff what I like is

⏹️ ▶️ ^Marco neutrality in most cases so for instance I don’t use a tube

⏹️ ▶️ ^Marco amp because I use sensitive headphones and sensitive headphones you can very easily pick

⏹️ ▶️ ^Marco up the noise introduced by tube amps you can also occasionally get like a little pop as if a piece of dust

⏹️ ▶️ ^Marco hits the wrong way or something like that that like it’s just not it is it’s it’s

⏹️ ▶️ ^Marco almost like like records you know it’s almost like vinyl it’s not you’re introducing distortion

⏹️ ▶️ ^Marco you’re introducing noise so I try to get very very nice digital

⏹️ ▶️ ^Marco stuff so everything is solid state there’s there’s you know no moving parts of the volume knob

⏹️ ▶️ ^Marco there is as little noise introduced as possible and

⏹️ ▶️ ^Marco there is a little you know tonal imbalance introduced as possible and I

⏹️ ▶️ ^Marco actually took it a little too far. So my my current setup is I have the Sennheiser

⏹️ ▶️ ^Marco HD 800 headphones plugged into the shit Asgard 2 that is

⏹️ ▶️ ^Marco actually its name and the shit Bifrost DAC

⏹️ ▶️ ^Marco and it looks amazing these things all look fancy and awesome they sound great

⏹️ ▶️ ^Marco the main reason I like the shit amp and deck is because they they have like zero noise

⏹️ ▶️ ^Marco that I can hear anything that I have

⏹️ ▶️ ^Casey it’s not because they’re the shit

⏹️ ▶️ ^Marco well and the name helps too and they look pretty awesome and they’re they’re small and they were reasonably priced

⏹️ ▶️ ^Marco and I like it because it’s super neutral now the Sennheiser HD 800 headphones

⏹️ ▶️ ^Marco the best thing I can say about them is that they are the most neutral headphones I’ve ever heard the downside of this

⏹️ ▶️ ^Marco is that they actually sound a little bit boring on certain things like certain types of music

⏹️ ▶️ ^Marco I actually like the distortion by things like amplifying the high-end

⏹️ ▶️ ^Marco that you hear in some biodynamic models or or the more the bigger bass I

⏹️ ▶️ ^Marco have on my closed headphones the th900s and certain things I like

⏹️ ▶️ ^Marco that distortion now throughout throughout all my research and finding these headphones actually I tried a few other models

⏹️ ▶️ ^Marco I ended up returning or selling in the meantime and I found that people’s opinions vary

⏹️ ▶️ ^Marco dramatically on what they considered sounding good. So one of the most

⏹️ ▶️ ^Marco popular lines the Sennheiser HD 600 and 650 those actually have pretty muted

⏹️ ▶️ ^Marco high end and I didn’t like that sound a lot of people loved

⏹️ ▶️ ^Marco it and similarly like certain people don’t like the biodynamic super strong

⏹️ ▶️ ^Marco high end and I like it and what I found is that you know

⏹️ ▶️ ^Marco keeping everything neutral up until the point of the actual headphones allow me to make that choice gives

⏹️ ▶️ ^Marco me the flexibility like you can simulate all the other you can simulate vinyl with digital

⏹️ ▶️ ^Marco plugins and equalizers and stuff please email Dan

⏹️ ▶️ ^Marco you can simulate most of the most of the effects of what people like about these things

⏹️ ▶️ ^Marco electronically artificially if you want to just like Instagram you know can

⏹️ ▶️ ^Marco you know can simulate the the distortions of certain old processes or things that

⏹️ ▶️ ^Marco you know or like idealized or enhanced versions of old processes and old distortions that used to exist in the analog

⏹️ ▶️ ^Marco world and that produces more pleasing looking photos just to a lot of people. So it’s important to recognize

⏹️ ▶️ ^Marco like you know I like I like everything to be digital and pure and clean

⏹️ ▶️ ^Marco and neutral up until the point where I have my headphones where I can make that choice and say either I want to listen to something with a lot

⏹️ ▶️ ^Marco of bass with this pair of headphones or I want to listen to something that’s neutral with this pair of headphones and And I recognize

⏹️ ▶️ ^Marco that’s a crazy setup, however, that’s how I am and that’s how I did it. But that, you know, none of that, like the argument

⏹️ ▶️ ^Marco John was making last week is that CDs are better at being

⏹️ ▶️ ^Marco neutral. They’re better at accurately and neutrally replicating the recording.

⏹️ ▶️ ^Marco And that’s, that is correct. That is 100% right. And

⏹️ ▶️ ^Marco most people, though, when they’re arguing about what sounds better or worse or what is better or worse, they

⏹️ ▶️ ^Marco don’t put it in those terms of personal preference or of like the difference between neutrality

⏹️ ▶️ ^Marco and pleasing sound. Because those are very different things. Perfectly neutral sounds

⏹️ ▶️ ^Marco boring to a lot of people. Like when I edit our podcast, I edit with perfectly neutral. However,

⏹️ ▶️ ^Marco when I listen to Phish, I like a little bit more bass. So when I listen to Phish, I put on the other headphones. You know,

⏹️ ▶️ ^Marco it’s very important to distinguish those things. preference or more pleasing versus

⏹️ ▶️ ^Marco accurate and neutral.

⏹️ ▶️ ^Casey Yeah, I think you’re right, but I don’t know. This is probably getting a bit old now.

⏹️ ▶️ ^Casey Why don’t you tell us about something that’s cool?

⏹️ ▶️ ^Marco I would love to. It’s our friends at PDFPen. Once again, this is PDFPen for iPad.

⏹️ ▶️ ^Marco PDFPen for iPad lets you edit PDFs anywhere you are, as long as you have an iPad with you that happens

⏹️ ▶️ ^Marco to be running PDFPen. However, anywhere you are with one of those things, you can edit PDFs. You can sign

⏹️ ▶️ ^Marco contracts, you can fix a typo, you can actually edit the PDF right there on your device,

⏹️ ▶️ ^Marco you can correct a price list, you can fill out a form while you’re on the go, and you can take PDF documents

⏹️ ▶️ ^Marco with you anytime, add notes, highlighting, add markup during your mobile downtime.

⏹️ ▶️ ^Marco You can sync PDFs with PDFPen for OS X using iCloud or Dropbox, because

⏹️ ▶️ ^Marco PDFPen is for OS X and iPad. You can grab and save PDFs using Dropbox, Evernote,

⏹️ ▶️ ^Marco Google Drive, Box, I guess it isn’t called Box.net anymore? Is that right? They dropped the.net, now it’s just Box?

⏹️ ▶️ ^Marco Anyway, you can save PDFs to that or from that, even WebDAV and FTP

⏹️ ▶️ ^Marco servers. And anywhere you are with PDFPen for iPad, you have the complete

⏹️ ▶️ ^Marco feature-rich mobile editing power of PDFPen. It’s available in the App Store. Now,

⏹️ ▶️ ^Marco a lot of apps these days, I’ve been meaning to talk about this, a lot of apps these days haven’t even been updated for iOS 7 yet, they’re

⏹️ ▶️ ^Marco not native, they’re kind of ignored. PDF pen is awesome, it is always updated,

⏹️ ▶️ ^Marco it is native to iOS 7, and there’s tons of… they’re always making tons

⏹️ ▶️ ^Marco of improvements. Performance improvements, everything else, they’re fantastic. And it syncs all

⏹️ ▶️ ^Marco your stuff, once again. They even support a few of the pressure-sensitive pens on the market,

⏹️ ▶️ ^Marco the most popular ones. They support the Jot Touch, the Pogo Connect, and the Jaja

⏹️ ▶️ ^Marco or HaHa, I’m not sure which one as that is, but it’s J-A-J-A, that one, which is either Jaja or Haha

⏹️ ▶️ ^Marco or maybe Jaha or Haja. Who knows? But they support that one. So check out PDFPen

⏹️ ▶️ ^Marco for iPad. Go to smilesoftware.com slash ATP. That is

⏹️ ▶️ ^Marco smilesoftware.com slash ATP. Thanks a lot to PDFPen for iPad for sponsoring our show

⏹️ ▶️ ^Marco once again. I’ll tell you, I use PDFPen and it’s fantastic. I

⏹️ ▶️ ^Marco use it more on the Mac than the iPad just because that’s where I am most of the time. But it’s, boy, It’s an amazing piece of software and

⏹️ ▶️ ^Marco it saves my butt pretty frequently. So thanks a lot to PDFPen for iPad once again.

⏹️ ▶️ ^Casey We didn’t get a chance to talk about last week because we were too busy arguing.

⏹️ ▶️ ^Casey We didn’t get a chance to talk about this trial with Samsung, between Samsung and Apple.

⏹️ ▶️ ^Casey And there’s been some interesting things coming out of this. And we’ve gotten to look

⏹️ ▶️ ^Casey at some of the behind the scenes at how Apple does stuff,

⏹️ ▶️ ^Casey how they get their jobs done as a result of a lot of the discovery during

⏹️ ▶️ ^Casey the trial. So I don’t know, John, do you have any thoughts on this?

⏹️ ▶️ ^John John Greenewald Yeah, I picked out in particular, not so much the details of the trial or whatever, because isn’t this one

⏹️ ▶️ ^John have to do with patents too? I don’t pay attention, but you

⏹️ ▶️ ^{John, Casey} know, I think so

⏹️ ▶️ ^John about patents or whatever. But yeah, because as part of the discovery process, or some other legal term

⏹️ ▶️ ^John that I don’t know, they’re revealing internal documents from Apple.

⏹️ ▶️ ^John It’s great because Apple is, well, it’s great for us anyway. I don’t know if it’s great for Apple. That’s a whole other discussion. But

⏹️ ▶️ ^John we get so little insight into what goes on in Apple, we just see their public face. And their public face is

⏹️ ▶️ ^John so incredibly controlled. And I think it’s even more controlled now than it used to be, because as we’ve said many times

⏹️ ▶️ ^John before, in the good old days with Steve Jobs, yes, Apple’s message was controlled. But Steve Jobs

⏹️ ▶️ ^John had the ability and often took it to just go off book and do whatever the hell he

⏹️ ▶️ ^John wants and say whatever the hell he wants in whatever way he wants. Sure, he was, you know, controlled and calculated as well, but you

⏹️ ▶️ ^John never quite knew what he was gonna say. Tim Cook, you’re not gonna make him slip up and say something. You’re not gonna get

⏹️ ▶️ ^John something exciting. So far, it seems like he is very on message and

⏹️ ▶️ ^John very controlled, strategically leaking things and saying things in a certain way and choosing his words very carefully.

⏹️ ▶️ ^John But with Steve Jobs, you felt like there was a chance he might, you know, either act like he’s just

⏹️ ▶️ ^John suddenly caught up in the moment and saying something spontaneous or actually do that. But anyway, these

⏹️ ▶️ ^John documents are internal Apple documents. And from the outside, a lot of people say Apple is arrogant,

⏹️ ▶️ ^John Apple’s out of touch, they don’t know what their users really want or whatever. But these internal documents make it very, very

⏹️ ▶️ ^John clear that that is not the case, that it is merely their external face that they control so well. But internally,

⏹️ ▶️ ^John this is the fiscal year 2014 planning offsite slide deck. And you can see

⏹️ ▶️ ^John some of these slides, if you haven’t seen them, we’ll put the link in the show notes, you should check them out. But I’ll read off

⏹️ ▶️ ^John some of the slides here. So the first slide says, they’re talking about iPhone sales, and it says, growth rates are slowing.

⏹️ ▶️ ^John And they show a graph that shows growth rates going down, down, down, down, down. Apple, of course, would never show

⏹️ ▶️ ^John a slide like this in public, because why are they going to show something that puts them

⏹️ ▶️ ^John in a bad light? And again, something as simple as this, showing growth

⏹️ ▶️ ^John rates slowing. Yes, you’ll see stories about Apple about that. And if someone actually gets in touch with Apple and asks them about that, they’ll say some

⏹️ ▶️ ^John blah, blah, blah, kind of, well, we think we’re on the right track and blah, blah, blah.

⏹️ ▶️ ^John They’ll say some stupid PR-y thing. But here they are saying, when they’re talking to themselves, they’re like, here’s the graph. Growth rate is

⏹️ ▶️ ^John really slowing. Next slide, they say, so what’s going on? And they have little sections here that says,

⏹️ ▶️ ^John strongest demand is coming from less expensive and larger screen smartphones. Carriers have strong

⏹️ ▶️ ^John interest in capping iPhone due to one or more factors. High share, the subsidy premium, unfriendly

⏹️ ▶️ ^John policy, unfriendly is in quotes, lack of alignment. So they’re basically saying, customers want cheaper

⏹️ ▶️ ^John phones with bigger screens. Carriers, carriers don’t like us that much. And they don’t like us for good reasons, because we are

⏹️ ▶️ ^John more demanding of them, because these unfriendly policies that we have, our subsidy is really high,

⏹️ ▶️ ^John lack of alignment with, you know, our interests aren’t aligned with the carriers. You know, they’re

⏹️ ▶️ ^John basically laying out, yeah, carriers don’t like us that much. And competitors, and they show the little Android icon. Competitors have drastically

⏹️ ▶️ ^John improved their hardware and software and their ecosystems. And spending

⏹️ ▶️ ^John obscene, also in scare quotes, amounts of money on advertising and carrier channel to gain traction, which is

⏹️ ▶️ ^John a Samsung thing. So laying out, like, here are our problems. We’re kind of getting our butt kicked in these areas.

⏹️ ▶️ ^John Here’s what people want. And this is the title of the slide is great. Next slide. Consumers

⏹️ ▶️ ^John want what we don’t have. If you can imagine Apple saying that in public, what

⏹️ ▶️ ^John we don’t have. And so they show like the market share growth and they said, where

⏹️ ▶️ ^John did the growth come from? And this, I wish there was a better breakdown and maybe there was, but they’re saying,

⏹️ ▶️ ^John where did the growth in the industry come from? And it shows the growth and it says, I

⏹️ ▶️ ^John can’t do the percentages here, but more than half the growth came from phones that are less than $300. And

⏹️ ▶️ ^John the rest of the growth came from screens that are more than $300, but have screens that are bigger than four inches.

⏹️ ▶️ ^John The objection I have to this graph is, okay, great, you have identified that the majority of the growth

⏹️ ▶️ ^John is coming from phones that are less than $300. I think you need to break that down more,

⏹️ ▶️ ^John because if you take that piece and say, Okay, well, out of that growth, actually 99%

⏹️ ▶️ ^John of that growth is coming from phones that are less than $5. That is way different than saying, okay, well,

⏹️ ▶️ ^John most of it’s coming from, how much less than $300? Break it down. Is it, is it, you know, what’s the

⏹️ ▶️ ^John average? What’s the distribution in there? Without knowing that, you don’t know what to do. But anyway, this is a high level slide.

⏹️ ▶️ ^John So I take, I assume they have broken it down internally. I just, again, I would like to see that broken down. I didn’t.

⏹️ ▶️ ^John And all this reminded me of a book I’m reading now, which I would recommend to everybody and we’ll put a link in the show notes.

⏹️ ▶️ ^John It’s Ed Catmull’s book called Creativity, Inc. It’s not really a

⏹️ ▶️ ^John memoir or manifesto. It’s more of just an explanation of the early history of Pixar and

⏹️ ▶️ ^John what Ed Catmull and the rest of the company has done to make that organization

⏹️ ▶️ ^John function and be successful. And a big focus on the book is being clear-eyed

⏹️ ▶️ ^John about your own problems and candid about them internally. And I know Pixar’s not Apple, and I

⏹️ ▶️ ^John know Steve Jobs didn’t have nearly as much involvement in Pixar as he did in Apple. But

⏹️ ▶️ ^John I can’t help but see parallels between the things that are in

⏹️ ▶️ ^John this book and the approach Apple is taking here. Apple is very clear-eyed in these slides. They are not sugarcoating

⏹️ ▶️ ^John it. They are not like rah, rah, rah, like, yeah, we may have problems, but we still have the best phone in this

⏹️ ▶️ ^John office. The slides are just, you know, there’s not a lot of them, but they’re essentially unrelentingly negative

⏹️ ▶️ ^John towards Apple. And that’s how they have to be. That’s how Apple improves. So

⏹️ ▶️ ^John I felt better about seeing these slides, and it made me think that at least some of the culture and

⏹️ ▶️ ^John strategies described in that excellent book by Ed Catmull are alive and well in

⏹️ ▶️ ^John Apple as well.

⏹️ ▶️ ^Marco Yeah, I think it’s fantastic to see that Apple knows its own problems.

⏹️ ▶️ ^Marco It knows its own shortcomings. But I think that’s not really that

⏹️ ▶️ ^Marco much news, necessarily. To me, seeing all these

⏹️ ▶️ ^Marco trial documents and the emails of Phil Schiller and everything, it really just says to me

⏹️ ▶️ ^Marco that this company is, by being as private as they usually are,

⏹️ ▶️ ^Marco they are saving us from the boredom and the relative

⏹️ ▶️ ^Marco routineness that’s actually in their company. These are like, the only big surprise

⏹️ ▶️ ^Marco here is that these people are actually acting somewhat human but they’re all you know they’re saying not

⏹️ ▶️ ^Marco that surprising things they’re sending boring business emails to their advertising agency that doesn’t even capitalize

⏹️ ▶️ ^Marco anything which is apparently a thing.

⏹️ ▶️ ^John Well that was that was a whole other thing like the Phil Schiller like this article was supposed to be you know sensational or whatever

⏹️ ▶️ ^John and so one of the things they showed was look at this email where Phil Schiller is being angry to his ad agency what I took away from

⏹️ ▶️ ^John that exchange was how incredibly low the bar is for Apple’s ad agency

⏹️ ▶️ ^John like you feel like this is your ad agency you’re the biggest company in the world and these jokers

⏹️ ▶️ ^John are doing your ads boy that’s just depressing like I not everything can be up

⏹️ ▶️ ^John to the standard of Apple and Pixar I guess and it’s just yeah

⏹️ ▶️ ^Marco yeah I mean why what I really just found with these with these documents is like wow this is all

⏹️ ▶️ ^Marco really boring stuff that like it like there’s very little

⏹️ ▶️ ^Marco of newsworthiness here except that you know Apple usually does a really good job of keeping

⏹️ ▶️ ^Marco what comes out interesting you know, keeping their public image interesting and secretive

⏹️ ▶️ ^Marco and well edited. And you see now on the inside that yeah,

⏹️ ▶️ ^Marco it’s a big company dealing with big company problems and doing things that big companies do.

⏹️ ▶️ ^John It might not seem that novel to you but I compare it to, I mean, I work for a lot of different companies

⏹️ ▶️ ^John but the one people may have heard of is that for a time I worked for Palm and I didn’t, I wasn’t very close

⏹️ ▶️ ^John to the leadership of that organization but internal to the company when I worked for Palm I got presentations

⏹️ ▶️ ^John from, you know, some of the bigwigs came to visit us because we are in a different location and gave us a presentation on the state

⏹️ ▶️ ^John of Palm and what their plans were for the business. And their slides were not candid and didn’t accurately

⏹️ ▶️ ^John identify the problems they were facing. They were much more kind of like, we’re actually still doing

⏹️ ▶️ ^John pretty good in the PDA space and the phones are coming on, but we’ve got these trio things. And like, it was like,

⏹️ ▶️ ^John even to me inside the company, I’m like, you guys are in denial of these are the slides we’re showing this internally. If we are,

⏹️ ▶️ ^John we are showing these slides to ourselves internally. We are not being honest with ourselves about the state.

⏹️ ▶️ ^John I mean, you would hope that they would, you know, we saw it and we felt it. Maybe they felt it. It just didn’t feel

⏹️ ▶️ ^John as honest and straightforward. And like, if you if you don’t think these Apple slides are novel or interesting,

⏹️ ▶️ ^John like I don’t have that much experience in big tech companies, but I have seen it from big and small.

⏹️ ▶️ ^John Most companies are not this candid with themselves and are not are not this accurate, like, you know, are

⏹️ ▶️ ^John not able to exactly pinpoint where their weak points are and nail them in that way.

⏹️ ▶️ ^John There’s confidence internally and skill, and if they weren’t that skilled, they wouldn’t be where they are today.

⏹️ ▶️ ^John I think there is something to it. We shouldn’t just accept that every company does this because I think many of them don’t.

⏹️ ▶️ ^Casey The other thing that we didn’t talk about last week, which we could have, is some

⏹️ ▶️ ^Casey guy that apparently I was supposed to know but didn’t know is leaving Apple. That

⏹️ ▶️ ^Casey guy is Greg Christie. Had either of you two heard of him before? I’m assuming, John, that you had.

⏹️ ▶️ ^Casey But Marco, have you heard of this gentleman before last week?

⏹️ ▶️ ^Marco Nope. I still don’t even believe this is really a story.

⏹️ ▶️ ^John Well, there was there were stories about him. He was featured in recent stories before the he was leaving Apple

⏹️ ▶️ ^John story. So even if you weren’t aware of him before that, you understand. I’ve never seen that name before. It’s like, yes, like two weeks ago,

⏹️ ▶️ ^John you did see that name when it was I think someone did a story about the early days of the iPhone and he was heavily.

⏹️ ▶️ ^{John, Casey} Yeah, yeah.

⏹️ ▶️ ^John Yeah. So that’s why you might be aware of him. Yeah, this looks a lot like kind of a gossip story, like people

⏹️ ▶️ ^John arguing inside Apple and And then one of them leaves. I mean, that happens all the time in every company. And

⏹️ ▶️ ^John then, you know, from some people have, we don’t know what’s really going on here,

⏹️ ▶️ ^John but there’s at least one reasonable design denial from Matt Panzerino. The details of this big breakup

⏹️ ▶️ ^John are not as dramatic as they seem. But the reason I think it’s interesting is

⏹️ ▶️ ^John forget about how we left or why he left or whatever. Like, who cares? Right. The

⏹️ ▶️ ^John the what it highlights, what all these stories highlight is it’s time for them to talk about kind

⏹️ ▶️ ^John of, oh, Johnny Ives organization, because he is in charge of much more than he used to be in charge

⏹️ ▶️ ^John of anything that happens in that organization, people coming, people going or whatever,

⏹️ ▶️ ^John with him remaining the top, in some ways, like, not that it reflects on him, but it makes us

⏹️ ▶️ ^John focus on him. So what these stories made me do is think more about how Johnny

⏹️ ▶️ ^John Ives is doing in his position. And yes, part of his position is retention or whatever, but not so much in this particular

⏹️ ▶️ ^John But what it got me thinking about was what what are Johnny Ives weaknesses

⏹️ ▶️ ^John in his new role as software designer? I’ve had decades to think about what his weaknesses are in hardware design and

⏹️ ▶️ ^John previously was unknown what is he going to be like in software. But his organization, I guess, you know,

⏹️ ▶️ ^John iOS 7 is the first big product of his software organization and we’ve all had a long time

⏹️ ▶️ ^John to live with it. And I think his weaknesses as a software designer are

⏹️ ▶️ ^John more glaring and obvious than they are hardware maybe, and in fact, they’re very similar. And so anytime anything

⏹️ ▶️ ^John happens within that organization, it makes me sort of like, dwell more

⏹️ ▶️ ^John on how he’s doing in his new role as software designer.

⏹️ ▶️ ^John And I think that’s a useful avenue of thought for me.

⏹️ ▶️ ^Casey So are you not pleased with how things are going then with Johnny at the helm?

⏹️ ▶️ ^John Well, like, when somebody leaves like that, even if it’s on good terms, bad terms, or whatever,

⏹️ ▶️ ^John you have to think that it’s not Ive that’s leaving. People under him are going to leave. And you have to think,

⏹️ ▶️ ^John maybe there was a, it’s someone who used to be more in charge than they are now.

⏹️ ▶️ ^John He was largely responsible for a lot of the early iPhone UI. And surely his power

⏹️ ▶️ ^John to make changes and decisions has decreased now that Johnny Ive has come in above him. And you know what I mean?

⏹️ ▶️ ^John That’s the whole thing with the reorg. Johnny Ive has been given much more power. And that means the power,

⏹️ ▶️ ^John the people who used to have that have had it taken away. And if there’s any sort of disagreement about what

⏹️ ▶️ ^John to do, Johnny Abb’s going to win, as he should, because he’s in charge. And what that means is that whatever his

⏹️ ▶️ ^John strengths and weaknesses are, are imprinted even more strongly on the organization, because he is not just one voice

⏹️ ▶️ ^John anymore. He is the voice. As I think I tweeted at one point, he’s in full Shigeru Miyamoto

⏹️ ▶️ ^John mode, where Miyamoto designed Nintendo’s flagship games like Mario and

⏹️ ▶️ ^John Donkey Kong and many other things, and slowly climbed the ranks of the organization until essentially he was

⏹️ ▶️ ^John the buckstop with him for all issues related to software design and all of Nintendo and

⏹️ ▶️ ^John possibly several aspects of hardware design as well. Like, once you are in that position

⏹️ ▶️ ^John of power, whatever your strengths or weaknesses are magnified. I mean, it was true with Steve Jobs,

⏹️ ▶️ ^John his strengths and weaknesses were magnified through the whole organization. And so now that’s why I’m dwelling on what I was like

⏹️ ▶️ ^John as a software designer, because these other voices, whether you agree with them or not, Scott Frostall, Greg Christie,

⏹️ ▶️ ^John to other people, their voices and their ability to, you know, affect change

⏹️ ▶️ ^John within the organization are necessarily modified by the larger and larger footprint of this one fallible individual,

⏹️ ▶️ ^John for better or for worse. And so that’s what I’ve got my eye on.

⏹️ ▶️ ^Marco Yeah, that’s interesting. I mean, I, you know, the problem is, I, I think you’re right,

⏹️ ▶️ ^Marco that that is something to watch. But I think everything we’ve everything we’ve heard about this about Greg

⏹️ ▶️ ^Marco Christie’s particular departure makes it sound like maybe, you know, maybe

⏹️ ▶️ ^Marco the problem had nothing to do with Johnny Ive. And, you know, like we see like the

⏹️ ▶️ ^Marco crazy rumor sites saying, you know, there was some kind of friction there. And then you see people like John Gruber,

⏹️ ▶️ ^Marco who have generally better sources on these kind of things, saying, oh, this was actually,

⏹️ ▶️ ^Marco you know, in the works for months. And he didn’t even, this wasn’t even recent and there was no bad

⏹️ ▶️ ^Marco blood he just was retiring and as the story gets

⏹️ ▶️ ^Marco older we’re hearing more that it’s not a story than the

⏹️ ▶️ ^Marco very minute that it broke in the news that said it was a story so that’s probably the reality

⏹️ ▶️ ^Marco it probably is not a story in itself now again you’re right that this is something you know Johnny Ive having

⏹️ ▶️ ^Marco a lot more power than he used to and becoming so powerful that he’s able to get

⏹️ ▶️ ^Marco bad decisions through, that is something to watch, no question. And iOS 7

⏹️ ▶️ ^Marco is a great example of some of those things. I think iOS 7 really showed

⏹️ ▶️ ^Marco that Johnny Ive was wanting to make a big change

⏹️ ▶️ ^Marco in a lot of very important stuff on iOS, and he did, and he went a little too far in a lot

⏹️ ▶️ ^Marco of those changes. But even, as we saw last summer, even from beta 1 to

⏹️ ▶️ ^Marco what was released as 7.0 there were a lot of changes to reign some of that back like that the biggest

⏹️ ▶️ ^Marco one being the super thin font was replaced with a more reasonably thick thicknessed one

⏹️ ▶️ ^Marco more reasonably weighted one you know things like that I think there were the end the

⏹️ ▶️ ^Marco lock screen having modifications to make it easier to realize that you have to swipe right and not up

⏹️ ▶️ ^Marco things like that you know there were a lot of or is it left I I don’t know, anyway.

⏹️ ▶️ ^Marco There were a lot of changes made. You know, he went too far and then he dialed it back. And

⏹️ ▶️ ^Marco I think iOS 7 is great. I think the design of it is great. It is not perfect, but it is

⏹️ ▶️ ^Marco great. And it is a massive improvement over 6. So, and this is gonna get into,

⏹️ ▶️ ^Marco you know, what we want from iOS 8, I guess, but I don’t know if we even have that as a topic. But

⏹️ ▶️ ^Marco again, I think this is something to watch for, you know, watch for Johnny Ive getting enough power that he

⏹️ ▶️ ^Marco can get bad decisions pushed through. and for him to think that these decisions

⏹️ ▶️ ^Marco are good. You know, those things are a problem together. But other than that, I don’t really

⏹️ ▶️ ^Marco think this is much of a story.

⏹️ ▶️ ^John Like, on this particular story, it doesn’t, like, what I was getting at is it doesn’t matter how or why Greg Christie left. It only

⏹️ ▶️ ^John matters that he was somebody who was there for a long time, who was responsible

⏹️ ▶️ ^John for a lot of, like, he was a senior person, right? He wasn’t just some other person. He was another voice,

⏹️ ▶️ ^John whether he agreed with Johnny Ive or not at any point. If you are like the last

⏹️ ▶️ ^John Godzilla standing, like all the other big guys have left, all the other people who were there

⏹️ ▶️ ^John from the beginning who strongly influenced the original iPhone, you know what I mean? Like,

⏹️ ▶️ ^John that’s what I was saying about like Shigeru Miyamoto, like you need to have those voices that

⏹️ ▶️ ^John you will listen to because you believe like that they are valuable and have experience. And even if you just left

⏹️ ▶️ ^John because he just felt like retiring, right? That’s a loss. That’s somebody that you don’t have in the

⏹️ ▶️ ^John organization with a wealth of experience for you to bounce your ideas off of. And you know what I mean? Like, I’m going to

⏹️ ▶️ ^John get back to the Ed Catmull book again, but they have this whole chapter on the brain trust, which maybe we’ll put a link to this, because I think they did an excerpt

⏹️ ▶️ ^John on some website somewhere, where they will get the people, the sort of the oldest and wisest

⏹️ ▶️ ^John people together in a room and have them all tell each other what’s wrong with their

⏹️ ▶️ ^John projects just to bounce ideas. Because they all respect each other, and they all have a lot of experience.

⏹️ ▶️ ^John And any time a senior person leaves in the similar

⏹️ ▶️ ^John role, like the senior software designer for iOS in the past, that is

⏹️ ▶️ ^John a loss for the company. And that’s why I get like, it doesn’t matter who cares why he left,

⏹️ ▶️ ^John he’s not there anymore. And that will only serve to magnify the good and

⏹️ ▶️ ^John the bad about Ive. And like, I would rather see him surrounded by

⏹️ ▶️ ^John more senior people that he respected to not to temper

⏹️ ▶️ ^John his output, but perhaps even to enhance the parts that are good about it. I don’t like they have to disagree with him. Maybe they should more vehemently

⏹️ ▶️ ^John agree with him in certain areas where he’s doubting himself. Like that dynamic is what I what I think is lost when

⏹️ ▶️ ^John senior people leave.

⏹️ ▶️ ^Casey Yeah. The other thing that I’m a little concerned about just on based completely on theory

⏹️ ▶️ ^Casey is Johnny getting spread a little thin and I would hope that he has his

⏹️ ▶️ ^Casey trusted minions slash you know, advisors or whatever to

⏹️ ▶️ ^Casey to take a lot of the day to day off of his plate. But

⏹️ ▶️ ^Casey based on no facts and just a bunch of assumptions, it seems to me that Johnny is a fairly hands-on

⏹️ ▶️ ^Casey guy. And if he’s a hands-on guy and presumably already had a

⏹️ ▶️ ^Casey full plate with hardware alone, I just can’t imagine now adding all of software

⏹️ ▶️ ^Casey onto that already overflowing plate and still being able to

⏹️ ▶️ ^Casey be good at your job, good at not only your existing job of hardware, but but also good at this new

⏹️ ▶️ ^Casey job of software, which really, in the strictest sense, he doesn’t have any real

⏹️ ▶️ ^Casey experience or education in. So that kind of concerns me a bit, that he’ll be spread too

⏹️ ▶️ ^Casey thin over the next few years.

⏹️ ▶️ ^John Well, see, that distinction, if you were to ask Johnny Ive, I’m gonna play Johnny Ive now because I read a book about him

⏹️ ▶️ ^John once. We had on the podcast, I bet what he would say is, this distinction

⏹️ ▶️ ^John between hardware and software is an artificial one that has no bearing on the experience of using the product. It’s all one and

⏹️ ▶️ ^John the same part of the same product. And it makes perfect sense for them to be under the same umbrella, because

⏹️ ▶️ ^John that separation and that sort of arguing between them, I guess there is a tension technically, but that doesn’t matter to

⏹️ ▶️ ^John the end product. It has to be one thing. I think it’s perfectly valid to have that

⏹️ ▶️ ^John holistic approach to the product. And Apple has always had it. It’s just that now it’s literally embodied

⏹️ ▶️ ^John within one person. And yes, there’s a danger of that, and that maybe there was healthy tension between hardware and software. But I don’t worry

⏹️ ▶️ ^John about him being spread too thin, because I think he is fulfilling sort of the Steve Jobs type role

⏹️ ▶️ ^John of set the direction, be tastemaker, give thumbs up, thumbs down. He’s not in there

⏹️ ▶️ ^John drawing you eyes one at a time or whatever. Like, you know what I mean? He’s got a staff.

⏹️ ▶️ ^John He’s got people who do that. But in the end, the buck stops with him. And if he really wants it to be a super skinny font on a white

⏹️ ▶️ ^John background, that’s what it’s going to be. And like Marco said, if you have to dial

⏹️ ▶️ ^John it back, fine. In many ways, it’s better to go too big and then scale it back than to be too timid. You know what

⏹️ ▶️ ^John I mean? So I’m not giving him a bad grade or anything. I’m just watching it

⏹️ ▶️ ^John because I would rather see ideas have to fight for their life

⏹️ ▶️ ^John as hard as possible inside Apple and only the best ones make it out than to slide

⏹️ ▶️ ^John towards a situation where there’s not enough people with skills even

⏹️ ▶️ ^John close to Johnny Ive for him to do good work in the software area. In the hardware area, I don’t know what’s going

⏹️ ▶️ ^John on back there and we don’t know those people as well. And I think he has a trusted team that doesn’t have a lot of turnover. But in the software

⏹️ ▶️ ^John area, he’s lost Scott Forstall through what we think actually was a disagreement. And

⏹️ ▶️ ^John Greg Christie is now gone. Like, I would be happier if both of those people

⏹️ ▶️ ^John were still there and working in concert with Johnny Ive. But you know, what can you do?

⏹️ ▶️ ^Marco Well, you know, maybe the hardware stuff will just be a lot less work now because you can just look at like what HP and Samsung

⏹️ ▶️ ^Marco are making and just do stuff that looks like that, right?

⏹️ ▶️ ^John I think it’s the opposite. It doesn’t work the other way.

⏹️ ▶️ ^Casey You know, another thing that I should point out, and I am not through listening to it yet, but on

⏹️ ▶️ ^Casey Guy English’s and Renee Ritchie’s really great podcast, Debug, on episode 33

⏹️ ▶️ ^Casey – we’ll put a link to that in the show notes – they had Ken Faryon, who did – apparently was kind of

⏹️ ▶️ ^Casey the primary guy behind auto layout. And I’m only about two-thirds of the way through the episode, but it’s really,

⏹️ ▶️ ^Casey really good. One of the things that Ken talks about is kind of what it’s

⏹️ ▶️ ^Casey like to get an idea through Apple. You know, so he,

⏹️ ▶️ ^Casey I didn’t hear the auto layout specific part yet, but just for the sake of argument, you know,

⏹️ ▶️ ^Casey I go in there, I come up with the idea for auto layout, and apparently

⏹️ ▶️ ^Casey he basically only has to convince his immediate supervisor, or at

⏹️ ▶️ ^Casey least that’s what I gleaned from it. And I just thought that was really, really interesting that it’s not very

⏹️ ▶️ ^Casey bureaucratic. It’s not that you have to convince 3000 people all the way up the chain. And I’m sure

⏹️ ▶️ ^Casey that that is the case sometimes. But generally, for

⏹️ ▶️ ^Casey some things, you can just convince one man or woman, and that’s all it takes. And I just thought that was fascinating.

⏹️ ▶️ ^Marco We are also sponsored this week, once again, by New Relic. New Relic is an all-in-one

⏹️ ▶️ ^Marco web application performance management tool. Apparently there’s an acronym for that and it’s APM.

⏹️ ▶️ ^Marco New Relic lets you see performance from the end user experience through servers down to each line of your server

⏹️ ▶️ ^Marco side code. New Relic asks us to take a minute and say a big thank you to all the data

⏹️ ▶️ ^Marco nerds out there building all this great stuff that we all know and love. They would like to send a shout out to developers,

⏹️ ▶️ ^Marco software geeks, code jockeys, to the brave few who see things differently. Here’s to working nights,

⏹️ ▶️ ^Marco to wearing oversized concentration enhancing headphones upon your furrowed brows. New Relic thanks

⏹️ ▶️ ^Marco you, the entire internet thanks you. Nowadays if you’re in any business, you’re in the software business.

⏹️ ▶️ ^Marco Software powers our apps, runs our databases, manages our accounts, and runs e-commerce sites and email programs.

⏹️ ▶️ ^Marco When software breaks, everyone loses. New Relic helps improve your software performance,

⏹️ ▶️ ^Marco so your users have a better experience and your business is more successful. How’s that for a win-win?

⏹️ ▶️ ^Marco New Relic monitors every move your application makes across the entire stack. can show you what’s

⏹️ ▶️ ^Marco happening right now in your app. You can zero in on problems quickly with transaction tracing,

⏹️ ▶️ ^Marco SQL and no SQL performance analytics, application topology mapping, and deployment

⏹️ ▶️ ^Marco history markers and comparisons. Go to newrelic.com slash ATP

⏹️ ▶️ ^Marco and you can sign up for a free 30-day trial. Then all you got to do is deploy their agent

⏹️ ▶️ ^Marco on your server. It natively supports Ruby, PHP, Java,.NET, Python, and even

⏹️ ▶️ ^Marco Node.js. It’ll start collecting data immediately. And then you can quickly see inside your app to

⏹️ ▶️ ^Marco start finding hotspots, fixing issues, and optimizing performance. Once again, go to

⏹️ ▶️ ^Marco newrelic.com slash ATP to start monitoring your web apps performance today and finding

⏹️ ▶️ ^Marco all your issues and fixing them all. newrelic.com slash ATP. Thanks a lot.

⏹️ ▶️ ^Casey So one quick remaining thought on the Christie thing which I came up with. We

⏹️ ▶️ ^Casey said Christie had been there since the beginning. Let’s suppose hypothetically

⏹️ ▶️ ^Casey that he was pretty well aligned or pretty chummy with Forstall, then

⏹️ ▶️ ^Casey wouldn’t it make sense for him to either choose to leave or

⏹️ ▶️ ^Casey be asked to leave? And the quiet rumblings

⏹️ ▶️ ^Casey I’ve heard are that that’s exactly the case, that he was kind

⏹️ ▶️ ^Casey of in Forstall’s camp and maybe the person who’s replacing

⏹️ ▶️ ^Casey him is someone who likes to argue a little bit more. And if that’s the case, I’d consider that a wonderful

⏹️ ▶️ ^Casey thing. I think it’s very smart of people who are in a position of power to have

⏹️ ▶️ ^Casey someone around them that argues because it forces you to really,

⏹️ ▶️ ^Casey really vehemently believe in your own opinion or change it if the case

⏹️ ▶️ ^Casey may be.

⏹️ ▶️ ^John The best example of that from the Steve Jobs’ history is the, you know, him being

⏹️ ▶️ ^John adamant that iTunes, was it iTunes? The iPod shouldn’t be available for Windows, iTunes Store shouldn’t

⏹️ ▶️ ^John be available for Windows. Come on, you guys got to help me out here, he’s not remembering. But anyway, yeah, iTunes for

⏹️ ▶️ ^John Windows, the chat room says, so I will trust them. But anyway, whatever it was, it’s something that eventually came to pass and was an

⏹️ ▶️ ^John enormously positive thing. And Steve Jobs was dead set against it. And it was just all of his most

⏹️ ▶️ ^John trusted lieutenants were constantly on him, arguing just like would not give it up. Like, we’ve got to do iTunes for Windows. He

⏹️ ▶️ ^John said, no, I’m never going to do it. I mean, eventually, they wore him down. He’s like, fine, do frigging iTunes for Windows,

⏹️ ▶️ ^John whatever you want. That sounds like an unhealthy dynamic, but I think that is actually

⏹️ ▶️ ^John a company. I mean, it’s not the healthiest it could be. It would be better if, again, it was

⏹️ ▶️ ^John something more like the Pixar Brain Trust. But in many ways, it was similar, because with

⏹️ ▶️ ^John the Brain Trust thing, it’s a bunch of senior people. I’ll tell you what’s

⏹️ ▶️ ^John wrong with your thing. But none of them have the power to tell you what you have to do. Like, they can’t make

⏹️ ▶️ ^John you change your movie, in the case of the Braid. They just tell you, like, we think this is wrong, that’s wrong, and maybe they suggest solutions. But

⏹️ ▶️ ^John they have no authority to make you do any particular changes. And in the case of Steve Jobs, his lieutenants

⏹️ ▶️ ^John didn’t have the authority to make him put iTunes on Windows. But because he respected their

⏹️ ▶️ ^John opinions, and because they were so insistent on it eventually, he went against his own better judgment

⏹️ ▶️ ^John and said, you know what, what am I even paying these people for if not to advise me? and they are so strong and unanimous

⏹️ ▶️ ^John in their insistence that this is a good idea. He was kind of a baby about

⏹️ ▶️ ^John it and said, fine, whatever. Put your stupid thing on Windows if accounts are to be believed. And that

⏹️ ▶️ ^John definitely sounds like him. But in the end, the system worked.

⏹️ ▶️ ^Casey Yeah, and that’s exactly my point. And so I mean, whether or not Christie was forced

⏹️ ▶️ ^Casey out, depending on who’s replacing him, it could be a really wonderful thing. So we’ll see.

⏹️ ▶️ ^Casey All right, so we have a few other things we can talk about and something that

⏹️ ▶️ ^Casey we’ve had on the show notes for a while that I don’t think I completely understand yet, and

⏹️ ▶️ ^Casey I’d love to hear, John, you explain it to me, what is this P cell thing all about?

⏹️ ▶️ ^John Oh, goody. Now I get to explain another thing that I don’t know the details of, but I do think I know enough. I do

⏹️ ▶️ ^John know enough, I think, to explain the broad strokes, and the broad strokes are

⏹️ ▶️ ^John actually interesting. P-Sal is from Steve Pearlman is his name, I think.

⏹️ ▶️ ^John Inventor, one of the original inventors of QuickTime. He also did the onLive remote gaming service.

⏹️ ▶️ ^John And this is another one of his babies. And he has a little bit of reputation as kind of a genius crackpot, sort of like Stephen

⏹️ ▶️ ^John Wolfram. Undeniably, incredibly smart and doing amazing things, but also prone

⏹️ ▶️ ^John to presenting his creations as the best thing since sliced bread

⏹️ ▶️ ^John and kind of magic. And that’s the difficulty with P-Sal is, I forget what the original name for it was, but it’s been

⏹️ ▶️ ^John rebranded to P-Cell. It’s presented in a way that doesn’t

⏹️ ▶️ ^John explain how it works unless you already know how it works. And so it seems like magic. It’s like, you have all these problems

⏹️ ▶️ ^John with cell phone reception because there’s so many people with cell phones, and there’s so many towers, and there’s

⏹️ ▶️ ^John so much interference, and there’s weak signals and strong signals, and you can’t have the cell towers too close. Like, he’s basically describing what

⏹️ ▶️ ^John we all know about current cell phone reception. He’s like, well, what if that could all go away and you wouldn’t have any problem?

⏹️ ▶️ ^John Every single cell phone user would You have the full bandwidth of the entire tower. And you’re like, well, that’s impossible.

⏹️ ▶️ ^John What are you talking about? And it sounds like a perpetual motion machine, and you tend to dismiss him out of hand.

⏹️ ▶️ ^John Unless you already know what it is, in which case, you would think, oh, well, I know what that is. And then it becomes boring.

⏹️ ▶️ ^John And those are the two extremes. Like, this sounds like magic. I’m going to be like the only person on Earth. It’s like as if I have

⏹️ ▶️ ^John this whole cell tower to myself. And then you find out how it really works. And if you were in this

⏹️ ▶️ ^John field, it’s not anything novel or amazing or new, although there certain aspects of it that are that

⏹️ ▶️ ^John are interesting, from an engineering perspective, and you’re like, oh, it’s that, okay, well,

⏹️ ▶️ ^John now, now, I think I’m less excited about it. And that was my experience as well. So here’s

⏹️ ▶️ ^John what it is. What do you in the demonstrations, what he’s talking about is having self towers,

⏹️ ▶️ ^John and you can’t put them too close together, because they make interference with each other. And of course, every single cell phone that’s trying to communicate with self

⏹️ ▶️ ^John our cell tower is interfering with the other cell phones that are communicating with the cell tower. And there’s many different strategies

⏹️ ▶️ ^John that we use to allow multiple cell phones to talk to the tower. And you don’t need to know the details of all

⏹️ ▶️ ^John of them. But the

⏹️ ▶️ ^{John, Casey} most

⏹️ ▶️ ^John ancient one is like, let’s just all take turns. There’s different ways of multiplexing our signals

⏹️ ▶️ ^John with frequency division, where we carve up it into different channels, different frequencies,

⏹️ ▶️ ^John or code division multiplexing, where we send a signal that each thing can decode to figure out which part of the signal was relevant to it.

⏹️ ▶️ ^John But it all has to do with just a big shared space. And all those techniques, interference is bad because it screws up

⏹️ ▶️ ^John with the signal each thing is sending out and receiving. What P-Cell is trying to do

⏹️ ▶️ ^John is make it so that all the cell towers are aware

⏹️ ▶️ ^John of where all the cell phones are. And they’re aware of what signal is being put out by

⏹️ ▶️ ^John all the phones and all the other towers. And they do a whole bunch of really complicated math to figure out, OK,

⏹️ ▶️ ^John we know what we want to send everybody here. We’re going to output something such that our interference

⏹️ ▶️ ^John overlaps with each other in such a way that it sort of forms a hotspot

⏹️ ▶️ ^John right in the target area where all the combination of the interference from all the signals we’re putting out

⏹️ ▶️ ^John sum up to exactly what we want to send to that phone. And then all the interference from all the signals we’re putting out sum

⏹️ ▶️ ^John up to exactly the signal we want to put out to that other phone. And that’s why it’s like you get all of you get all the bandwidth, because

⏹️ ▶️ ^John it’s a bunch of things working in concert to figure out exactly what they need to send out so that the sum of all

⏹️ ▶️ ^John their crazy interference exactly equals the exact clean signal they want to send to that particular

⏹️ ▶️ ^John phone. I’m massively simplifying it, obviously. But this is sort of the upshot of how

⏹️ ▶️ ^John it works. And the innovative thing they have is, well, how do you do that? How can you have all these cell

⏹️ ▶️ ^John towers figuring out exactly what they need to put out sort of in real time, you know,

⏹️ ▶️ ^John all the signals that are being sent and received to figure out what they have to put out to make the interference overlap to hit every single little

⏹️ ▶️ ^John cell phone? That sounds crazy. How could you do that? Well, that’s part of the engineering breakthrough and that they claim to have a way to do

⏹️ ▶️ ^John that with fiber optic cables or whatever running between the cell towers, or slower links if you’re willing to

⏹️ ▶️ ^John allow for a little more slop in the system. And to be able to have computing

⏹️ ▶️ ^John in each one of those locations that scales linearly, which is another topic on the topic list that we may get to

⏹️ ▶️ ^John someday, scaling, that scales linearly. So it’s like, well, OK, so it’s easy to do for two receivers, but

⏹️ ▶️ ^John is it 10 times as much to do four? It is like, how does it scale? It’s like, well, we have a way, if you

⏹️ ▶️ ^John 50 people on a tower, then you need one computer. If you want to handle 100 people, you need two computers. 200, you need

⏹️ ▶️ ^John four computers. It scales linearly with the number of people. That is an interesting engineering breakthrough,

⏹️ ▶️ ^John assuming it works as advertised. But there are limitations to the system. First of all, you do have to have

⏹️ ▶️ ^John all the towers talking to each other so they can figure out what the hell to output. You do need the fast connections

⏹️ ▶️ ^John between the towers. And if you make a slower connection between the towers, what you’re giving up is essentially

⏹️ ▶️ ^John how fast the receivers can move. So for example, if you’re in a car

⏹️ ▶️ ^John and that car goes over something like 70 miles an hour, by the time all the cell towers figure out how

⏹️ ▶️ ^John to constructively overlay all their crazy overlapping signals to hit where you are, you’ve moved too far and

⏹️ ▶️ ^John it won’t work as well. So there are speed limitations to this, which are not, you know, there are speed limitations that speed,

⏹️ ▶️ ^John you moving around really quickly affects everything, but it affects this system more than others. So, you know, if you’re in any

⏹️ ▶️ ^John sort of car trying to, you know, use your nav system or whatever and the system was in use. If you’re going over 70, maybe it wouldn’t

⏹️ ▶️ ^John be able to keep up with you if you only had microwave links between the towers and everything. But the idea,

⏹️ ▶️ ^John and even I think Apple’s routers and regular just Wi-Fi routers do this, they call

⏹️ ▶️ ^John it beamforming or whatever. The idea is an interesting one. And it’s great when an idea that

⏹️ ▶️ ^John worked in labs for years and years suddenly becomes feasible in the real world due to engineering

⏹️ ▶️ ^John expertise, which is essentially what they’re trying to bring as a company to this. And the theoretical benefits

⏹️ ▶️ ^John are there, and that if you could get a system like this, you can make a much more effective use of bandwidth, and you can put small

⏹️ ▶️ ^John cell tower type emitters, receivers everywhere. And don’t worry about how they overlap, because

⏹️ ▶️ ^John it’ll be just fine. That would make for the ability to support more users in a

⏹️ ▶️ ^John congested urban environment. And

⏹️ ▶️ ^John the other trick they have from engineering perspective is they didn’t want to have to change all our phones. So they found a way using the existing

⏹️ ▶️ ^John features of LTE, 4G that are in the existing protocol, so such that an unmodified, plain

⏹️ ▶️ ^John old iPhone 5S, or any other plain old cell phone, can participate in this. Like, they don’t need

⏹️ ▶️ ^John to have a special chip in the phones, or special computing in the phones, or whatever. And that, again, is another engineering

⏹️ ▶️ ^John type of breakthrough where, yeah, this was fine in theory, but you can’t just go replace all the cell phones in the world with these magic ones that work.

⏹️ ▶️ ^John It’s like, well, we found a way to get this to work using existing protocols with existing unmodified phones with all the smarts

⏹️ ▶️ ^John on the other end. So I think it is an interesting engineering achievement, And assuming it

⏹️ ▶️ ^John works as advertised, he still faces go-to-market challenges in terms of getting this technology into all

⏹️ ▶️ ^John the cell towers and who’s going to be the first person to roll it out and how will it coexist with other things. But in the

⏹️ ▶️ ^John end, it is not magic. It’s just science. It’s not particularly new science, but the engineering

⏹️ ▶️ ^John is where the interesting parts

⏹️ ▶️ ^Casey are. There was a really neat video that I watched that showed, I believe

⏹️ ▶️ ^Casey it was a bunch of like laptops or tablets or whatever, that were equipped with LTE radios.

⏹️ ▶️ ^Casey And they’d shown, I’m probably get the details wrong, but they’d shown what

⏹️ ▶️ ^Casey happens when you move around, and they turned off the thing where it

⏹️ ▶️ ^Casey senses where you are. So basically, they concentrated this beam of LTE

⏹️ ▶️ ^Casey in a specific spot. And just for the purposes of the demonstration, rather than allow

⏹️ ▶️ ^Casey that beam to move around with the device, they kept it stationary and you could see

⏹️ ▶️ ^Casey as they move just a couple of inches that the throughput would just plummet

⏹️ ▶️ ^Casey and then they would put the device back where it started and it would come back to like full HD video

⏹️ ▶️ ^Casey or something like that. And then when they had it working as it’s designed, which is

⏹️ ▶️ ^Casey to say it will follow the device as it moves, you would get this full HD signal as the dudes like, you know,

⏹️ ▶️ ^Casey just moving these devices around in the room with a reasonable quickness, not 70 miles

⏹️ ▶️ ^Casey an hour, but a reasonable quickness. And it was fascinating to watch and really, really impressive.

⏹️ ▶️ ^John In some ways, it’s like engineering at its best, because what engineers do very well,

⏹️ ▶️ ^John as opposed to, you know, scientists, like, because they’re, you know, two sides of the same coin, as the scientists

⏹️ ▶️ ^John will come up with something or these theories or whatever. But it’s the job of the engineers to be able to figure out,

⏹️ ▶️ ^John you know what, that theory was useless to us 10 years ago, 20 years ago, 50 years ago.

⏹️ ▶️ ^John But computing has advanced so much, or technology, or material science, or whatever has advanced so much that that crazy theory that

⏹️ ▶️ ^John no one ever pays attention to, that it’s like, well, that’s not feasible. Obviously, you can never do that,

⏹️ ▶️ ^John starts to enter the realm of feasibility. And engineers are rewarded by getting to jump at everyone and say, you

⏹️ ▶️ ^John know what? I think I can do that now. I think I can do that crazy beam forming thing across

⏹️ ▶️ ^John huge urban areas to people traveling up to 70 miles an hour with existing. I think I can do that now. Like, to get

⏹️ ▶️ ^John there first, right? Because eventually, when technology comes along, and it’s like, well, now we have the, you know, everyone will realize

⏹️ ▶️ ^John we have the computing to do, you know, an algorithm that wasn’t feasible before, or a compression technique that used

⏹️ ▶️ ^John to be too computationally expensive, but now is trivial on phones, right? Like, and this is an example

⏹️ ▶️ ^John of that. Like, he seems to be out ahead of people in terms of, I think I can do that now, and here’s

⏹️ ▶️ ^John my proof, and here’s my implementation. And whether or not it flies, kind of like on live, like, I think I

⏹️ ▶️ ^John can make people hit video games where the video gaming machine is in the data center, and they are miles and miles away. I think

⏹️ ▶️ ^John gaming can work like that. It did kind of work. I have on live. I’ve

⏹️ ▶️ ^John launched it. I’ve played it. It’s not great, but as a technology proof of concept, it’s interesting.

⏹️ ▶️ ^John As with anything else, the engineering is only one small part of the battle of being successful in the market.

⏹️ ▶️ ^John But as non-magical as it might be,

⏹️ ▶️ ^John I do like what he does because kind of like Elon Musk, He’s

⏹️ ▶️ ^John figuring out what is feasible with current technology and getting there just slightly ahead of everyone else.

⏹️ ▶️ ^John And that’s great engineering.

⏹️ ▶️ ^Casey Yeah, like I said, it was really impressive to me whether or not it’s just

⏹️ ▶️ ^Casey the combination of a bunch of things we’ve known how to do for a while. It looked really

⏹️ ▶️ ^Casey cool.

⏹️ ▶️ ^John That’s all Apple does too, right? Take things, you know, we’ve had touchscreens, we had capacitive touchscreens,

⏹️ ▶️ ^John we’ve had user interfaces with buttons, like bring it all together, figure out that actually we can make a phone like

⏹️ ▶️ ^John that now and do it just a little bit ahead of everyone else. There’s a big reward if you get all the parts right.

⏹️ ▶️ ^Marco Exactly. We are also sponsored this week by Igloo software.

⏹️ ▶️ ^Marco Igloo is an intranet you’ll actually like. Now most people think of intranets as…

⏹️ ▶️ ^Marco remember this is not internets, this is intranets, the old boring thing, right? The old, stale, terrible

⏹️ ▶️ ^Marco places that the corporate overlords make you go when you just want to use Dropbox or WordPress or something

⏹️ ▶️ ^Marco easy that helps you get your worked on that actually works. Igloo brings the ease of use and familiarity

⏹️ ▶️ ^Marco from consumer software into your corporate environment by using familiar apps like shared calendars, twitter-like

⏹️ ▶️ ^Marco microblogs, file sharing, and more. Every piece of content in your igloo can be social,

⏹️ ▶️ ^Marco with comments and like buttons, and each team in your company can configure their own workspaces.

⏹️ ▶️ ^Marco So that’s it for your users, but what if you’re in charge of IT? Well igloo handles the security,

⏹️ ▶️ ^Marco hosting, and management for you. Igloo is SOC 2 type 2 compliant and they host

⏹️ ▶️ ^Marco data securely in SOC 2 type 1 enterprise facilities in Canada. That’s

⏹️ ▶️ ^Marco right, Canada, so everything is polite and CDRs cost too much, on their own servers.

⏹️ ▶️ ^Marco Igloo offers 256-bit SSL and they are not vulnerable to heart bleed. They offer

⏹️ ▶️ ^Marco backups, disaster recovery, single tenant and shared environments, integration with many authentication

⏹️ ▶️ ^Marco and sync systems, including SAML services and LDAP and more. You can even work

⏹️ ▶️ ^Marco with HIPAA compliant organizations. You can also customize everything inside your igloo with

⏹️ ▶️ ^Marco the ability to add CSS and JavaScript globally across one team space or even on a single

⏹️ ▶️ ^Marco page. You can see all this on the igloo website, which is actually built on their platform.

⏹️ ▶️ ^Marco Learn more at igloosoftware.com slash Casey.

⏹️ ▶️ ^{Marco, Casey} It’s

⏹️ ▶️ ^Marco igloosoftware.com slash Casey, where they made a funny landing page about why SharePoint sucks

⏹️ ▶️ ^Marco and how igloo is better. Thanks a lot once again to Alou for sponsoring our show.

⏹️ ▶️ ^Casey I wanted to ask about scaling servers because we’ve had this in the show notes for a while.

⏹️ ▶️ ^Marco I love this. So on the topic list, there’s just a simple bullet point that just says scaling servers.

⏹️ ▶️ ^Marco Like you might as well have had a bullet point that said computers.

⏹️ ▶️ ^John I wrote it there and it was actually a reaction to a particular blog post that Brent Simmons put up where

⏹️ ▶️ ^John he mentioned scaling and passing. And it’s not I don’t want to get into a big thing about it, but it’s just like

⏹️ ▶️ ^John every time I see this come up and we do it ourselves and people are sloppy with terminology like it’s so important

⏹️ ▶️ ^John to keep reminding yourself like, you know, definition of terms like what

⏹️ ▶️ ^John we should be talking about when we talk about scaling because we use it as an umbrella term to talk about everything. And I kind of alluded to

⏹️ ▶️ ^John it in the piece l thing, but like my only point is like the old saw about the difference between performance and scaling.

⏹️ ▶️ ^John And so much of the time when I see people thinking they’re blogging about scaling, all

⏹️ ▶️ ^John blogging about as performance, which is fine and interesting a topic of itself, but it’s not sending a scaling.

⏹️ ▶️ ^John And so I just, you know, I put it in there as a reminder to myself to throw it out. So in brief performance,

⏹️ ▶️ ^John how fast you can make something go. If you’ve got a computer and you’ve got a task to be done, I can do that task in five minutes.

⏹️ ▶️ ^John I need to increase my performance. Now I can do that task in three seconds. You’ve increased your performance. Scaling

⏹️ ▶️ ^John is I’ve got one task to do and I’ve got one computer. What if I give you 10 million of

⏹️ ▶️ ^John those tasks to do? Can you do them in the same time? If you just use 10 million computers.

⏹️ ▶️ ^John That’s scaling. It doesn’t matter how long it takes. Even if it takes your computer a year to solve the problem, if I give you two of those

⏹️ ▶️ ^John problems, you say no problem, I’ll just buy another computer. I can do both of them in a year. Okay, what if I give you three?

⏹️ ▶️ ^John I’ll buy three computers. This is linear scaling. I’ll buy three computers. I can do all three in a year. What if I give you three

⏹️ ▶️ ^John million? I’ll buy three million computers. I can do all three million of those problems in a year. It’s not about performance.

⏹️ ▶️ ^John Scaling and performance, although they often seem related, are not the same concept.

⏹️ ▶️ ^John The The failure to focus on scaling when you’re talking about scaling and performance when you’re talking about performance leads to

⏹️ ▶️ ^John all sorts of sadness.

⏹️ ▶️ ^Marco Oh yeah, I mean, scaling is not how fast your

⏹️ ▶️ ^Marco system is, but how easily you can make it take on more of something.

⏹️ ▶️ ^Marco Usually more traffic, more users, more data, whatever the case may be.

⏹️ ▶️ ^Marco How easy is it to take on more? And how many changes do you have to make to your software and to

⏹️ ▶️ ^Marco its architecture to add that capacity because you know it’s with

⏹️ ▶️ ^Marco almost everything it is not nearly as simple as oh just add more servers because you might have like all right

⏹️ ▶️ ^Marco well let’s say you know you can add more web servers pretty easily up to a point because web servers like

⏹️ ▶️ ^Marco you know you can distribute the calls to any of them with a load balancer as long as you don’t use like weird shared

⏹️ ▶️ ^Marco sessions or anything you can distribute calls to any of them there any call coming in gets assigned to web

⏹️ ▶️ ^Marco server 67 and web server 67 does whatever it needs to do and responds back. Okay, great.

⏹️ ▶️ ^Marco However, WebServer 67 probably has to access a database to do something.

⏹️ ▶️ ^Marco And how do you organize the databases? And that’s usually, it’s the data layer, not

⏹️ ▶️ ^Marco the web serving application layer, that’s usually the hardest to scale. Because

⏹️ ▶️ ^Marco that’s when you have things like globally shared storage. You have things like

⏹️ ▶️ ^Marco limits on disk performance and on how many How many writes can

⏹️ ▶️ ^Marco a single server even write per second? If you’ve scaled away all the

⏹️ ▶️ ^Marco reads and given reads to all these replicating servers, that’s fine, but

⏹️ ▶️ ^Marco can they even keep up with the writes? How many writes does the master do? If you split the roll

⏹️ ▶️ ^Marco up like that, then you get into really hard issues like, all right, well, how do you cache data?

⏹️ ▶️ ^Marco Can you be sure that the data that you have read off of the read slave is up to date?

⏹️ ▶️ ^Marco Otherwise, you can’t really cache it because you might be clobbering something in the cache that is newer. Or you might be caching

⏹️ ▶️ ^Marco old data when new data is available. And there’s all these like really hard problems that go along with it that makes

⏹️ ▶️ ^Marco scaling such a complex, interesting topic and why it’s such a hard problem.

⏹️ ▶️ ^John And that’s why when I see a blog post about like how to make a data structure smaller or a query

⏹️ ▶️ ^John more efficient, and then the word scaling using that same conversation, it’s like that’s performance. You’re just talking

⏹️ ▶️ ^John about performance. And you’re talking about the web tier, like, since that, you know, oh, it’s stateless. We can scale that

⏹️ ▶️ ^John horizontally forever. Like, nothing scale. Even the web tier, even the web tier is

⏹️ ▶️ ^John not like that one-to-one scaling that I was using in my example. If you ever get there, you have reached the golden, you know,

⏹️ ▶️ ^John that you will be a bazillionaire. Because even just getting there, even on the web tier, it’s like, well, but what about different data

⏹️ ▶️ ^John centers? And now we need a geographical load balancing. And what about network connections? Like, even nothing,

⏹️ ▶️ ^John nothing scales perfectly. But you’re right, like, you know, the first thing you run into it is with state. And you get into

⏹️ ▶️ ^John the CAP theorem of consistency, partition tolerance, and availability.

⏹️ ▶️ ^John This is well-trod territory. Lots of people will concentrate in scaling. Google knows probably the most about it of any company

⏹️ ▶️ ^John in the entire world, given the scale they operate at. Microsoft seems to be learning a lot about it. Apple, not sure what they

⏹️ ▶️ ^John know. But scaling is super hard. Performance is super hard.

⏹️ ▶️ ^John And I mean, from the outside, this is all sort of esoteric stuff.

⏹️ ▶️ ^John But even for people who are just working on a small system. And it’s like, it doesn’t matter. You’re

⏹️ ▶️ ^John not going to have Facebook level users, right? But even on a very small system, 10 users, 100 users, or 1,000, 200,000, even then,

⏹️ ▶️ ^John you should be thinking about both performance

⏹️ ▶️ ^John and scaling and keeping a clear head about which one you’re concentrating on. And don’t concentrate too much on scaling and not enough

⏹️ ▶️ ^John in performance or too much in performance and not enough scaling. Just keeping a clear head about what

⏹️ ▶️ ^John the individual tasks are. And you will eventually find yourself, even in a small system, reading

⏹️ ▶️ ^John all those papers about the theory behind it and making your tradeoffs about, you know,

⏹️ ▶️ ^John consistency, availability and partition tolerance, like you have to make decisions about if you don’t make decisions, the software you choose

⏹️ ▶️ ^John will make the decisions for you and you’ll be sad about it. So like, that’s all part of the learning process.

⏹️ ▶️ ^John But it’s, I don’t know, I, it’s I don’t want to pick on any particular

⏹️ ▶️ ^John blog posts. I love reading those Brent posts. I love watching him think out loud about what he’s doing. And he’s a great example of a great software

⏹️ ▶️ ^John developer, like sort of, You know showing his showing his work as he works

⏹️ ▶️ ^John towards a product and everything like that. It’s just that I Have since I’ve dealt with server-side software

⏹️ ▶️ ^John my entire career I have a particular itch when I see the word scaling and

⏹️ ▶️ ^John it’s not really talking about scaling and maybe I’m just being Obedient right Casey

⏹️ ▶️ ^Casey That’s right. That’s right.

⏹️ ▶️ ^Marco Well, and there is there is something to be said for performance optimization

⏹️ ▶️ ^Marco to a point. It’s very similar to the premature optimization wisdom in the world, except, and Brent even addressed

⏹️ ▶️ ^Marco this in one of his posts on Inessential.com, which we should mention. He even addressed,

⏹️ ▶️ ^Marco when you’re talking about server-side design, for client-side stuff,

⏹️ ▶️ ^Marco premature optimization is considered a bad thing because we have such fast hardware.

⏹️ ▶️ ^Marco We have so much memory these days on client-side devices and desktops and everything. We

⏹️ ▶️ ^Marco have so many resources that it’s just not really worth

⏹️ ▶️ ^Marco doing super micro-optimizations everywhere. However, on servers, things

⏹️ ▶️ ^Marco are a little bit different. On servers, you are paying for what you use, basically,

⏹️ ▶️ ^Marco and if your app gets popular at all, even for a minimal

⏹️ ▶️ ^Marco amount of users, relatively speaking to something like Facebook,

⏹️ ▶️ ^Marco if you get a hundred thousand users using a web service. That’s

⏹️ ▶️ ^Marco substantial needs for that hardware and depending on what your service is

⏹️ ▶️ ^Marco doing that actually could need some serious resources and so

⏹️ ▶️ ^Marco that could you know you could be talking the difference between hundreds of dollars a month

⏹️ ▶️ ^Marco or thousands of dollars a month worth of hardware or even more than that and so

⏹️ ▶️ ^Marco you know it can really make or break your whole economics it can it can make you busier or not as

⏹️ ▶️ ^Marco busy. One of the best things to happen for servers and scaling in the last decade

⏹️ ▶️ ^Marco has been SSDs. Because for a long time, for a lot of LAMP type applications,

⏹️ ▶️ ^Marco one of the biggest limitations was the database’s disk speed. And like a Tumblr,

⏹️ ▶️ ^Marco I was able to replace I think it was about 9 or 12 reed slave

⏹️ ▶️ ^Marco servers with one with SSDs and you know 9 or 12

⏹️ ▶️ ^Marco servers that had 15k SAS disks in raid 10 being replaced with

⏹️ ▶️ ^Marco one SSD base server and the and those were those were old Intel x25e SSDs like you

⏹️ ▶️ ^Marco know they’ve come a long way since then that you know that now modern SSDs are even faster than those

⏹️ ▶️ ^Marco and you know it’s like when when you can make a big order of magnitude

⏹️ ▶️ ^Marco type jump. If you can go from 10 database servers to one database server,

⏹️ ▶️ ^Marco that kind of difference actually matters and is worth doing. That is something where

⏹️ ▶️ ^Marco performance has a direct measurable, feelable impact on scaling.

⏹️ ▶️ ^Marco Optimization for performance to a point actually is useful because in practice,

⏹️ ▶️ ^Marco most people who make these web services and make apps and stuff like

⏹️ ▶️ ^Marco that, most people are not going to get as big as Facebook and Google. Most people are not

⏹️ ▶️ ^Marco even going to get as big as, you know, smaller things that we’ve all heard of, you know. It’s most people are lucky to

⏹️ ▶️ ^Marco get a hundred thousand users to their web service, you know. And so by making decisions

⏹️ ▶️ ^Marco that favor high performance, you can often keep, you know, that could often

⏹️ ▶️ ^Marco be the difference between being able to run everything off of a cheap, easy to

⏹️ ▶️ ^Marco administer VPS or a fully managed cloud service at a reasonable price,

⏹️ ▶️ ^Marco and running a whole bunch of dedicated servers or running or paying tons of money for a cloud

⏹️ ▶️ ^Marco service or one of these things like that could make that big of a difference. And so it is, or, you know, or

⏹️ ▶️ ^Marco on the scaling side, this could be the difference between, you know, needing just one fast database server and needing

⏹️ ▶️ ^Marco a multi server setup or being able to fit all on one server or having to shard the database,

⏹️ ▶️ ^Marco which is really complicated and adds a lot of maintenance or having to use a different type

⏹️ ▶️ ^Marco of tool entirely like Cassandra, which is a world of hurt. It’s just so incredibly not worth

⏹️ ▶️ ^Marco using. And so, performance does matter to

⏹️ ▶️ ^Marco the point where having a high-performance system has a very

⏹️ ▶️ ^Marco good chance of keeping you from ever needing a difficult-to-scale system.

⏹️ ▶️ ^John if they’re successful, well, the actual business, the people who don’t understand, you know, the business people

⏹️ ▶️ ^John who don’t understand technology and don’t want to will apply both of these pressures to you because

⏹️ ▶️ ^John they will apply pressure for you to increase performance because they don’t want to pay a lot of money for

⏹️ ▶️ ^John servers, right? But they will also apply the pressure of you for scaling because they will say, my business plan says

⏹️ ▶️ ^John we’re going to grow 50% year over year for the next five years. So you need to be able to say, okay,

⏹️ ▶️ ^John we have five customers now, in in five years, we’re going to have 5 million customers.

⏹️ ▶️ ^John So I, you know, whatever multiple that is, I don’t want to pay if we if you have

⏹️ ▶️ ^John greater than linear scaling, it’s like, okay, well, one customer costs us 10 bucks, but two customers costs $100, you’ve

⏹️ ▶️ ^John got a scaling problem immediately, because they want the business to grow. But they want it to grow less than linear,

⏹️ ▶️ ^John they don’t want it, they want to go to double their number of customers, but not double their hardware costs. And what you’re telling them as well,

⏹️ ▶️ ^John that’s not how scaling works. If we double our customers, I need to way more than double our hardware costs. And it’s like, how much more

⏹️ ▶️ ^John how close are you to perfect linear scaling? And so they want both they want

⏹️ ▶️ ^John it, you know, it’s too expensive. Every time we get another client, we need to buy x amount of hardware and it costs so much money. And

⏹️ ▶️ ^John we’re making only a certain amount of money from each individual client we had, we can’t afford to add that much hardware. So increase your performance.

⏹️ ▶️ ^John But year over year over year, they’re like, look, we’re gonna quadruple our customer base, can you quadruple our customer base? And you’re

⏹️ ▶️ ^John like, we can’t we got one database server, we’re already buying the biggest machine that we could possibly that

⏹️ ▶️ ^John money can buy for our database server, we need a strategy for scaling. And then you end up sharding.

⏹️ ▶️ ^John And what about people in different parts of the country? Then you evolve. If they’re in shards, how do these people interact with each other?

⏹️ ▶️ ^John And in some ways, it’s easier for software where it doesn’t matter, like Facebook, where you’re like,

⏹️ ▶️ ^John the message shows up here a little bit sooner than it shows up there. It’s not a banking system or whatever.

⏹️ ▶️ ^John But not everyone’s that lucky. Sometimes it really has to be. That’s where you get into the cap theorem again, where you have to pick

⏹️ ▶️ ^John your trade-offs. And it’s difficult. And even no matter how bounded your problem

⏹️ ▶️ ^John is, even it just all does a scale everything down. If you’re like, oh, I want

⏹️ ▶️ ^John to stick to, you know, VPS is or something really cheap. I don’t want to buy dedicated like you’re playing in the small leagues,

⏹️ ▶️ ^John but it still matters for your company when you’re going from 10 customers to 10,000. It’s the same

⏹️ ▶️ ^John exact thing playing out is just a smaller scale versus companies that start off from day one buying massive dedicated

⏹️ ▶️ ^John hardware. And the only difference in the high end end game is that the high end people are more

⏹️ ▶️ ^John likely to get into the situation where they are buying the most expensive computer equipment that money can buy from anyone in the entire

⏹️ ▶️ ^John planet. When they get to the end of their scaling rope, and they’re like, well, we’ve increased

⏹️ ▶️ ^John performance as much as we can go. No matter how much money we waive at somebody, we cannot get a bigger

⏹️ ▶️ ^John single machine to do task x. It’s like, now, if you waited this long to think about scaling, you

⏹️ ▶️ ^John have a serious problem, because you’ve hit the performance dead end. I guess you would optimize your software.

⏹️ ▶️ ^John like yeah it’s the same it’s the same story it’s just you know it’s a fractal of itself it just just

⏹️ ▶️ ^John depends on how many commas there are before the decimal point in the invoices

⏹️ ▶️ ^Marco and and that’s this is one of the reasons why I really prefer to do

⏹️ ▶️ ^Marco things at a relatively low level and to do things that are fairly standard because

⏹️ ▶️ ^Marco you know even if you don’t need scaling you know in quotes even even if you don’t think you need that today

⏹️ ▶️ ^Marco like knowing how to scale something or knowing the steps that you take

⏹️ ▶️ ^Marco and the hurdles you will face is useful when writing anything even before it needs

⏹️ ▶️ ^Marco to be scaled. So for instance, I’ve said on a number

⏹️ ▶️ ^Marco of occasions I never rewrite database joins, ever. Joins

⏹️ ▶️ ^Marco basically put a lot of work on the database server, which granted it is very optimized to do,

⏹️ ▶️ ^Marco however I prefer to split up the joins into two calls, like one to fetch the list of

⏹️ ▶️ ^Marco IDs and the second to fetch from the target table, you know, which ones you get and stuff like that and design

⏹️ ▶️ ^Marco the schema and the code to do that. And that’s for two major reasons. One

⏹️ ▶️ ^Marco is to make the to give the database a little bit less work. Well, three major reasons. One give

⏹️ ▶️ ^Marco the database less work because database CPU power database resources are expensive and hard to scale.

⏹️ ▶️ ^Marco The second one is caching. Maybe you can cache that first query, you know, that maybe

⏹️ ▶️ ^Marco you can cache the list of IDs you need and then, you know, only fetch the records or vice versa, only

⏹️ ▶️ ^Marco fetch the IDs and then fetch the records from cache, things like that. And then the third is that then that also gives me the ability

⏹️ ▶️ ^Marco later on, if the service gets really big to say, you know what, the users table

⏹️ ▶️ ^Marco now has to have its own database. This gets hit so often. And it’s so it’s such a drain of performance that

⏹️ ▶️ ^Marco we should split this up into its own database cluster entirely. So users

⏹️ ▶️ ^Marco is now on a different database, then you can’t do a join, because it’s not even the same server anymore. So

⏹️ ▶️ ^Marco even with Overcast, I’m not running any joins. I’m doing it the same way I’ve always done it because

⏹️ ▶️ ^Marco it’s not that much harder and it’s just in case in the future

⏹️ ▶️ ^Marco that’ll be fine, I’ll be set up for that, I’ll be ready to go. There’s so many other things, like

⏹️ ▶️ ^Marco very careful use of indexing. One of the most useful books I ever read

⏹️ ▶️ ^Marco on scaling was High Performance MySQL. I read the

⏹️ ▶️ ^Marco edition before the current one and I liked it a lot. The current one got a lot longer and the reviews weren’t

⏹️ ▶️ ^Marco as good so I’m not sure if the current one is great. I didn’t actually read the whole thing. But the previous one was awesome

⏹️ ▶️ ^Marco so maybe give it a shot. But I think it’s important

⏹️ ▶️ ^Marco to make those decisions. If you’re writing, suppose you’re writing C code for an app and

⏹️ ▶️ ^Marco you’re calling something like Sterlin in the loop to say like, iterate through this string

⏹️ ▶️ ^Marco and you’re calling Sterlin, I is less than sterling string. If you know about performance

⏹️ ▶️ ^Marco at all, you’re gonna look at that and say, wait a minute, I’m, you know, that’s being invoked on every single call.

⏹️ ▶️ ^Marco You get this mentality that makes you make small decisions like that better.

⏹️ ▶️ ^Marco And in the course of a whole project, that adds up. And at the end, even though any one of

⏹️ ▶️ ^Marco those calls probably won’t matter, when you make every decision with a certain mindset and a certain sensibility

⏹️ ▶️ ^Marco and a certain wisdom about what will happen in the future, or what the cost of this might be or might add

⏹️ ▶️ ^Marco up to. When you make all those small decisions the right way, the

⏹️ ▶️ ^Marco sum of all of that is different and does matter. Same thing applies to scaling. When you

⏹️ ▶️ ^Marco make a bunch of small decisions to say, oh you know what, I can put a little more work on the database here, or

⏹️ ▶️ ^Marco you know, this doesn’t need to be that optimized, even, you know, the index isn’t quite right to do this index only,

⏹️ ▶️ ^Marco or, you know, this is going to have to scan a bunch of rows to get these results, but

⏹️ ▶️ ^Marco that’ll be alright, you know, it’s, how How often do you really have to do that? Once you get into the mindset

⏹️ ▶️ ^Marco of thinking, okay, what’s going to happen when I have to do this thousands of times a second?

⏹️ ▶️ ^Marco You’ll probably never have to. But if you think about that from the beginning, you can make better

⏹️ ▶️ ^Marco small decisions along the way that will add up and that will make scaling easier for you.

⏹️ ▶️ ^Casey Can we go back a step? You’re saying that you’d rather make round

⏹️ ▶️ ^Casey trips to the database than do something that that the database is specifically designed to do.

⏹️ ▶️ ^John I mean. I think what he’s saying is that he doesn’t want to use a database.

⏹️ ▶️ ^Casey Right.

⏹️ ▶️ ^John I know you don’t like the newfangled key value storage things or whatever, but if you’re not going to do joins,

⏹️ ▶️ ^John and I’m not saying that’s the right or wrong decision, but if that’s the route you’re going, there are probably

⏹️ ▶️ ^John places you could store your data that would give you better performance for

⏹️ ▶️ ^John this same kind of usage pattern. I imagine the reason you wouldn’t like them because they’re newer

⏹️ ▶️ ^John and less mature, and you’re used to the features of being able to back up MySQL and do easy

⏹️ ▶️ ^John replication. And those are all important features and the reason to stick with it. But you’re kind of using MySQL in sort of a degenerate

⏹️ ▶️ ^John state. Like you’re using it as a really bad performing poor man’s key value storage

⏹️ ▶️ ^John with some basic filtering, doing your own joins client side. Which is fine, because you get the other benefits of it.

⏹️ ▶️ ^John Your familiarity, easy backup, easy replication, reliability. And that’s why you’re

⏹️ ▶️ ^John avoiding, that’s why you’re not another MongoDB nightmare story.

⏹️ ▶️ ^Marco Yeah, I think, obviously, I know, as you and as the chat have said, I’m going to get a lot of email about

⏹️ ▶️ ^Marco my no joins stance. But the fact is, first of

⏹️ ▶️ ^Marco all, you just said MySQL is way slower than some other system. And that’s probably not the case.

⏹️ ▶️ ^Marco Certainly it is not always the case. The reason I use MySQL, even though I’m not using

⏹️ ▶️ ^Marco some of the relational database type features like joins.

⏹️ ▶️ ^Marco I don’t use store procedures either, that’s another thing where that’s just asking for trouble, I don’t use triggers, things like that.

⏹️ ▶️ ^Marco The reason I do this is because MySQL is just freaking awesome.

⏹️ ▶️ ^Marco There’s no other way to say it. I’ve heard so much crap about MySQL from people who

⏹️ ▶️ ^Marco don’t use it and are prejudiced against it, whatever. And I know, I don’t use Postgres, I don’t use Oracle, I

⏹️ ▶️ ^Marco don’t have any experience with So I can’t tell you how it compares to those, but I can tell you most of the criticism

⏹️ ▶️ ^Marco I’ve heard about MySQL is wrong, or at best outdated and only applied to MyISAM

⏹️ ▶️ ^Marco and not the more recent InnoDB storage engine, which at this point is not even recent.

⏹️ ▶️ ^Marco And so if you use MySQL with InnoDB as your storage engine for all

⏹️ ▶️ ^Marco tables, it’s amazing. It is awesome. There it’s and I said this in a

⏹️ ▶️ ^Marco post recently and I you know the gravity of this, I don’t want to overstate this, but it’s it’s it’s hard to

⏹️ ▶️ ^Marco overstate this. In all the time at Tumblr, Instapaper, all that time, now

⏹️ ▶️ ^Marco Overcast, I have used MySQL a lot. And you know with Tumblr it was under extreme

⏹️ ▶️ ^Marco stress for the entire time I was working there. It was constantly under extreme stress.

⏹️ ▶️ ^Marco I have never seen MySQL crash once. I have never seen MySQL corrupt data

⏹️ ▶️ ^Marco once. That’s amazing! look at any… Well, you’ve never seen it corrupt data.

⏹️ ▶️ ^Marco I have never seen it corrupt data. I’ve never seen it crash. That’s…

⏹️ ▶️ ^John MySQL’s actual functionality could be argued to be a corruption of data. I know it’s documented.

⏹️ ▶️ ^John I know if you look it up in the docs, it says that’s exactly what it’s going to do. It’s not wrong. It’s not a bug.

⏹️ ▶️ ^John But that’s the philosophical difference that many quote unquote real database people have with MySQL. It’s

⏹️ ▶️ ^John like the documented behavior. when it performs that documented behavior, database people

⏹️ ▶️ ^John grit their teeth and get angry at it. And I’m kind of one of them.

⏹️ ▶️ ^Marco I’m not that familiar with exactly what you’re talking about. However, are you sure you’re talking about InnoDB and modern

⏹️ ▶️ ^Marco versions of this? No, no.

⏹️ ▶️ ^John Like weird coercions of data types and cases where columns that are marked as not null can

⏹️ ▶️ ^John actually have nulls and weird things with empty strings. They’re all documented. You go to one of those MySQL gotcha

⏹️ ▶️ ^John pages and just read through them. And in terms of performance, though, now that I’m saying you should switch

⏹️ ▶️ ^John to a different thing or something, but because most of the things that have massively better performance have much, much

⏹️ ▶️ ^John less of the other things you just described, like stability and reliability. But performance-wise,

⏹️ ▶️ ^John if your data model is kind of simple and never going to have joins,

⏹️ ▶️ ^John and if you care about getting more performance, which you may or may not, there are lots of other database systems that do go

⏹️ ▶️ ^John way faster than MySQL, especially within ODB. because it’s

⏹️ ▶️ ^John the extra overhead of that storage engine. Most of the new hotness

⏹️ ▶️ ^John is basically people do almost everything in memory. It’s

⏹️ ▶️ ^John not like Memcache, although there is a persistent Memcache thing, but a lot of them are like, look, we can’t involve disk

⏹️ ▶️ ^John in the operation of our thing here. And as you can imagine, when you do almost everything in memory, you

⏹️ ▶️ ^John go way faster than MySQL trying to do stuff with InnoDB.

⏹️ ▶️ ^Marco Oh, sure. Well, and there’s ways you can, You can take advantage of that with caching already. You can do, and by the way,

⏹️ ▶️ ^Marco MySQL does already cache a lot in memory, but you know, certain, and of course, the OS will cache a lot of the

⏹️ ▶️ ^Marco disk blocks memory and stuff like that, but you know, certainly, you can use MySQL with Redis

⏹️ ▶️ ^Marco or Memcache or any of the cache systems that are out there, and it’s fine, I mean, most people do, it’s fine.

⏹️ ▶️ ^Marco You know, that’s like people, there’s this big, it’s the cool thing to use a different

⏹️ ▶️ ^Marco storage engine these days, but the fact is, a storage engine is, oh boy, here we go. storage engine, it’s like a

⏹️ ▶️ ^Marco file system. In you know, you don’t want to change that very often. And you want it to be extremely

⏹️ ▶️ ^Marco conservative, and extremely focused on data integrity. And my SQL

⏹️ ▶️ ^Marco is so incredibly like battle tested. It is so solid. And yeah, I

⏹️ ▶️ ^Marco get that it might not have always been that way back in the olden days with my exam and all the crap, but those days are

⏹️ ▶️ ^Marco well into the past now. And modern my SQL, my SQL has been rock solid, I would say

⏹️ ▶️ ^Marco for at least seven or eight years and that’s why Google use it very heavily. I

⏹️ ▶️ ^Marco don’t know if they still do but they at least did. I know lots of major major sites

⏹️ ▶️ ^Marco use MySQL. Twitter use it for a long time and they probably still do. There’s a reason

⏹️ ▶️ ^Marco why. It’s really really good and it’s people there was this great

⏹️ ▶️ ^Marco blog post a few years back when FriendFeed existed. So to give you some idea

⏹️ ▶️ ^Marco and it was a blog post called how FriendFeed uses MySQL to store schema-less data and I’ll

⏹️ ▶️ ^Marco have to find it and put it in the show notes. But there’s a lot of good stuff in here. FriendFeed was a weird

⏹️ ▶️ ^Marco project that was run by incredibly good engineers and

⏹️ ▶️ ^Marco it basically addresses a lot of this stuff. Like MySQL, even if you don’t use some

⏹️ ▶️ ^Marco of the database-y features of it, MySQL is still an amazingly

⏹️ ▶️ ^Marco fast, solid, low-needs, low-maintenance storage engine that has

⏹️ ▶️ ^Marco tons of tools, a huge ecosystem, tons of optimization potential.

⏹️ ▶️ ^Marco There’s so many reasons to use it and if you look at one of the newer systems it’s

⏹️ ▶️ ^Marco you know and you know things like Mongo or you know all these like new no SQL type systems

⏹️ ▶️ ^Marco if you look at a lot of those yeah they have some benefits but if you don’t necessarily need those benefits and

⏹️ ▶️ ^Marco what you really want is for your database to be like a file system and to be basically bulletproof

⏹️ ▶️ ^Marco MySQL is a great choice and I feel like it gets ignored because it’s old.

⏹️ ▶️ ^John This comparison is kind of harsh so I don’t mean that the way it’s gonna sound but it’s it’s as close

⏹️ ▶️ ^John as we can align them if we’re forced to align these two things. Here we go. In the world of databases what

⏹️ ▶️ ^John is the PHP of databases? It’s kind of my SQL. It’s way better than PHP don’t get me wrong but if you have

⏹️ ▶️ ^John to map one to it like that’s what ends up in like what’s the Python of databases? That’s probably Postgres

⏹️ ▶️ ^John you know like that that’s that’s more or less how things line up and like what’s What’s the C, C++ of databases?

⏹️ ▶️ ^John That’s like Oracle. And what’s the Ruby? It’s like VoltDB or something.

⏹️ ▶️ ^John But to align things, I say very clearly, PHP, at my school, is the PHP of databases.

⏹️ ▶️ ^John And that sounds terrible, I know, because it is not as bad as PHP. Do not get me wrong. PHP is terrible. But

⏹️ ▶️ ^John it has many of the same characteristics in that it’s everywhere. It’s a known quantity. It’s reliable. It has

⏹️ ▶️ ^John idiosyncrasies. But once you know what those idiosyncrasies are, they don’t bother you that much. And the thing about using it for Schemaless,

⏹️ ▶️ ^John That’s why people do that, because they’re like, OK, I have a tool that I know is reliable. I know what it’s capable

⏹️ ▶️ ^John of. And I’m going to use it, like I said, in a degenerate kind of way. Like, I’m actually going to use it for schema-less data.

⏹️ ▶️ ^John I know it’s not a schema-less database system. I know there actually are schemas. But I’m going to define some table with columns for key and

⏹️ ▶️ ^John value and version number and some other crap. And it’ll work. And it’s like you are taking advantage

⏹️ ▶️ ^John of a system that works in a way that you’re comfortable with, that has all the features you need and is reliable.

⏹️ ▶️ ^John And you’re building something else on top of that. And all I was saying with the other database type things is like,

⏹️ ▶️ ^John if you care more about performance, there is these days much more performance to be had if you’re willing to trade it

⏹️ ▶️ ^John for less reliability and more unknowns. And I’m hoping that that sort of younger

⏹️ ▶️ ^John section of the data storage world matures and coalesces and we get some sort of equivalent

⏹️ ▶️ ^John of MySQL, or even best case some equivalent of Postgres, where it’s like a tried and true

⏹️ ▶️ ^John known quantity within that realm. Right now we don’t have it now. Right now it’s kind of the Cambrian explosion over there. And we’re not sure

⏹️ ▶️ ^John how it’s going to shake out. It’s still worth keeping your eye on because

⏹️ ▶️ ^John if, you know, 20 years from now, you’re still using MySQL with no joins, it will be either a great failure

⏹️ ▶️ ^John of the NoSQL world to come up with a useful product for the long term or a great failure of you to

⏹️ ▶️ ^John keep an eye on what’s going on over there.

⏹️ ▶️ ^Casey Possibly both. I have two questions. Firstly, what is the pearl of databases, John?

⏹️ ▶️ ^John Yeah, I was trying to think of that. I don’t know. Maybe, maybe MSQL, but neither one of you

⏹️ ▶️ ^John remember that. There really is no Pearl. If there is, I don’t know what it

⏹️ ▶️ ^John is. It’s probably one of the newfangled database systems that I’m just not familiar enough with.

⏹️ ▶️ ^John Because there’s nothing that has the same combination of a precursor

⏹️ ▶️ ^John to all of the better known ones now, with lots of weird quirks. I don’t know what that

⏹️ ▶️ ^John is. But it’s not MySQL, definitely.

⏹️ ▶️ ^Casey Isn’t it just some academic paper that swears that it’s academically

⏹️ ▶️ ^{Casey, John} flawless?

⏹️ ▶️ ^John That’s Pearl 6, you’re thinking of.

⏹️ ▶️ ^Marco a database that can be written but not read?

⏹️ ▶️ ^John PHB looks just like Perl. You can’t even make that comment. All

⏹️ ▶️ ^Casey right, let’s not go down that road. Can we still back up to you just spent, Marco, five or ten minutes explaining

⏹️ ▶️ ^Casey how MySQL is amazing and reliable and does what it’s supposed to do

⏹️ ▶️ ^Casey really well, but you don’t trust it to do a fricking join? I can’t get past

⏹️ ▶️ ^Marco that. No, no. It isn’t an issue of trust. It’s an issue of leaving yourself open for future scaling

⏹️ ▶️ ^Marco options. It’s about being able to say, you know what, my database server is going to

⏹️ ▶️ ^Marco at some point be split off from my web server once I want to have my service has more than five users. And

⏹️ ▶️ ^Marco this is going to have its own server and it’s going to be harder and more expensive to increase

⏹️ ▶️ ^Marco database hardware resources than to increase application or web hardware resources. And so

⏹️ ▶️ ^Marco you might as well have the application and web stuff do more of the work and make the database

⏹️ ▶️ ^Marco give the database an easier job. So things like, you know, reduce the number of queries to it, but the number of queries

⏹️ ▶️ ^Marco that go to it, that, you know, make them access fewer rows, make them access fewer tables,

⏹️ ▶️ ^Marco make it have to do less CPU work, but it’s more about IO, you know, let the database

⏹️ ▶️ ^Marco do only what the database is required to do, and let your other servers take on as much work as they can.

⏹️ ▶️ ^Casey OK, but you’d rather ping away at it and do a gazillion round

⏹️ ▶️ ^Casey trips to avoid these joints like I haven’t been through

⏹️ ▶️ ^Casey a Tumblr. I haven’t been through an Instapaper. So I’m doing that thing which I told you I

⏹️ ▶️ ^Casey don’t want to do in the beginning of the episode where I’m saying that seems weird. weird. But but genuine question,

⏹️ ▶️ ^Casey have you ever done anything to empirically prove to yourself? Like, have you used a new relic

⏹️ ▶️ ^Casey to prove to yourself that this is legitimately the right way to go?

⏹️ ▶️ ^John Like, you can do it on paper, I think you could convince yourself. But I’m sure I’m not gonna answer for Marco. But I’m sure the answer

⏹️ ▶️ ^John to that is yes. Because

⏹️ ▶️ ^{John, Marco} anyone

⏹️ ▶️ ^John who works in service at offer quickly realizes, oh, the part that has all the state is the hardest part to scale. I mean, he basically

⏹️ ▶️ ^John just laid it out, like you can just do it on paper, there’s a certain amount of work that needs to be done. One part of your system, you

⏹️ ▶️ ^John can scale really, really easily. One part is really hard to scale. You want the hard to scale part to do

⏹️ ▶️ ^John like the same, whatever operation does the most efficiently primary key lookups, maybe have it

⏹️ ▶️ ^John do that same operation over and over and over again, and have everything else you can do put on the easier

⏹️ ▶️ ^John to scale parts like you know, there’s lots of ways you can do this with math to figure out, you know how it works,

⏹️ ▶️ ^John but like just I think you even if you just think about that logically and reason through it. And

⏹️ ▶️ ^John that’s not even gonna get you linear scaling, like we’re just to keep our heads above water with this type of strategy. You always

⏹️ ▶️ ^John want to push the work out to the system that’s easiest to scale. And so,

⏹️ ▶️ ^John I mean, it’s kind of the Google approach or like with Bigtable where they went to the extreme where their data store was

⏹️ ▶️ ^John really annoying to deal with and unreliable. And applications had to retry and figure out how to resolve

⏹️ ▶️ ^John conflicts. And it was just like, it made it incredibly hard to write applications. But that was the price of scaling early

⏹️ ▶️ ^John on. And they’ve made strides in that way. But basically, you’re trying to move the work to the systems that are

⏹️ ▶️ ^John easy to scale. And web servers are way easier to scale, and not performance scale. And

⏹️ ▶️ ^John again, you’re confusing performance and like, oh, we’re going to take those round trips. You’re going across the network, you’re making multiple

⏹️ ▶️ ^John queries. It’s like, that’s performance. We’re talking about scaling.

⏹️ ▶️ ^Casey Yeah, that’s a fair point. And I did that deliberately.

⏹️ ▶️ ^Marco For whatever it’s worth, you know, we’re putting a lot of, you’re putting a lot of weight on these round trips.

⏹️ ▶️ ^Marco Round trips to a database server or to a memcache server that’s, you know, in the same rack

⏹️ ▶️ ^Marco or at least in the same data center as the

⏹️ ▶️ ^{Marco, Casey} server that you’re coming from. That’s a good point.

⏹️ ▶️ ^Marco You know, we’re not talking about a lot of time here. I don’t think… A lot of this depends on

⏹️ ▶️ ^Marco the kind of application you’re writing. How much data are you actually querying? To build the page or to build

⏹️ ▶️ ^Marco the API response that you’re talking about, how many database calls do you actually have to do? Is it

⏹️ ▶️ ^Marco seven or eight? Is it 50? Is it more? Or like there’s the what you’re doing

⏹️ ▶️ ^Marco matters a lot. And you know it’s Instapaper was fairly easy to scale

⏹️ ▶️ ^Marco because it just was never as big as Tumblr. And I was using I was using Tumblr style techniques at Instapaper.

⏹️ ▶️ ^Marco So of course it was way overkill. Which is great which is why which is why I was able to afford to

⏹️ ▶️ ^Marco run it and not go crazy and not go broke. But you know at

⏹️ ▶️ ^Marco Tumblr we face things like to give you one example. How do you display

⏹️ ▶️ ^Marco the list of posts on somebody’s dashboard? So you have to figure out who they follow

⏹️ ▶️ ^Marco and then of everyone they follow, find enough posts from those people to make 20

⏹️ ▶️ ^Marco of them and order them properly. There are so many different ways to do that. And there are the naive

⏹️ ▶️ ^Marco ways where you do a few joins, and then you do a big sort and it’s all fine.

⏹️ ▶️ ^Marco But you know, the naive way does not really scale very well, just because you start dealing with hundreds of

⏹️ ▶️ ^Marco gigs of data pretty quickly and the database having to scan millions

⏹️ ▶️ ^Marco of rows and that starts performing very badly. So a lot of times

⏹️ ▶️ ^Marco what you think is the most good, naive approach, actually in practice

⏹️ ▶️ ^Marco is not useful. It does not scale well enough. It becomes too expensive

⏹️ ▶️ ^Marco or becomes completely impractical or impossible to do at scale.

⏹️ ▶️ ^Marco So you start having to do weird little hacks. And so, you know, one of the hacks that we did at

⏹️ ▶️ ^Marco Tumblr early on was actually something that I learned from Wikipedia, what they did on their schema back,

⏹️ ▶️ ^Marco and this was probably 2007, 2008, which is you actually have

⏹️ ▶️ ^Marco basically a separate table that acts as an index.

⏹️ ▶️ ^Marco Because in certain ways, it uses memory more efficiently to do that, or in certain ways, the query optimizer will

⏹️ ▶️ ^Marco use that better. And so like you’ll have we had like a posts reference table that was basically

⏹️ ▶️ ^Marco the it was a certain it was a special index of the posts table and the post table was this

⏹️ ▶️ ^Marco massive thing that was hundreds of gigs that would only ever be accessed by primary key because as

⏹️ ▶️ ^Marco john said primary key access are very fast and everything’s optimized for that they cache very well everything else

⏹️ ▶️ ^Marco so you get the list of post IDs from other calls and then you could fetch cache posts

⏹️ ▶️ ^Marco or fetch post from the database or even get them from slaves and it was mostly all right. So there’s all sorts of things you can

⏹️ ▶️ ^Marco do if you leave yourself options. That’s what I’m saying. So and you know, look, I would love

⏹️ ▶️ ^Marco again, this is like the idea. It’s almost the final discussion. Academically, I would

⏹️ ▶️ ^Marco love to say, yeah, you know, just use the database as intended. And it works great. And that’s the easiest and fastest.

⏹️ ▶️ ^Marco In practice, it’s not. In practice, things are more complicated. And you can do the

⏹️ ▶️ ^Marco nice, simple way for a while. But eventually, you’re going to hit a wall.

⏹️ ▶️ ^Marco And what options you have when you hit that wall depends on how you’ve built the system and

⏹️ ▶️ ^Marco how much effort it is to get over that wall and how much it will cost you in both things like hardware

⏹️ ▶️ ^Marco or service costs and in administration time and maintenance over time.

⏹️ ▶️ ^Marco That will all depend on how you’ve built that system. And if you’ve built it to give yourself good scaling options

⏹️ ▶️ ^Marco in mind and to do things like be gentle on your databases,

⏹️ ▶️ ^Marco you will have more options and it will be cheaper and easier to scale.

⏹️ ▶️ ^John is an analogy that I think Casey will be able to identify with. It’s like, you know when you have like, well, maybe

⏹️ ▶️ ^John in C++ at least, when you have code and you want to make it faster, very often that involves making

⏹️ ▶️ ^John it longer and uglier. This is the same type of thing. You’re like, how could it be faster? I’m adding more and more lines

⏹️ ▶️ ^John of code. Like whether you’re unrolling a loop or doing some setting up some big bunch of setup stuff to do a bunch of

⏹️ ▶️ ^John SIMD operations on it, like it gets longer and it gets uglier. It looks like you’re doing more. It’s like, this seems like

⏹️ ▶️ ^John it’s more work. In the same way, you’re like, well, I had this nice query that got me all the data I wanted and one

⏹️ ▶️ ^John big self-consistent blob with this nice joint query, and I’m replacing it with these multiple queries when I just mesh

⏹️ ▶️ ^John the data together myself. Like, it’s longer. How could that be better? I mean, I guess it’s not

⏹️ ▶️ ^John fully accurate in the C, C++ case. Like, you’re actually increasing performance. And here, like Marco said,

⏹️ ▶️ ^John you’re protecting yourself for future scaling. Although in some cases, even in the deep, dark world of databases,

⏹️ ▶️ ^John you’ve been in a story procedure before, if your query planner is not your friend, like, say, if you use Informix,

⏹️ ▶️ ^John speaking from experience, sometimes if you want to get a massive amount of data, it’s better

⏹️ ▶️ ^John to write a stored procedure and essentially be the query planner yourself. Do it all in the database,

⏹️ ▶️ ^John all right on top of the thing, but say, look, when I give you this join, you’re going to do something dumb. So let me run this query, this query,

⏹️ ▶️ ^John this query, join the meshed results together, put them in a temp table, index the temp table, join

⏹️ ▶️ ^John against that temp table with the second thing. And you’re like, how could that possibly be faster than just running the query you wanted? It’s like,

⏹️ ▶️ ^John well, the query planner made some very unfortunate life choices when we sent it to Squary. As a matter of

⏹️ ▶️ ^John fact, doing this crazy stored procedure. Like, I mean, once you’re creating temp tables and indexing them, you’re like, there’s

⏹️ ▶️ ^John no way that could be faster than running that query. You’re like, well, let me show you. Like, databases are funny things.

⏹️ ▶️ ^John And the more you can treat them the way Marco’s treating them, the less headaches you’ll have

⏹️ ▶️ ^John about them. And I think the more you should think about whether you should even be using a database. But that’s,

⏹️ ▶️ ^John you know, that’s a conversation for another time. The idea that the sort of client-side

⏹️ ▶️ ^John code or the consuming code gets longer and uglier and more complicated, that is not outside

⏹️ ▶️ ^John the realm of normal programmer experience when just increasing the performance of your regular

⏹️ ▶️ ^John compiled code.

⏹️ ▶️ ^Casey Yeah, that makes sense.

⏹️ ▶️ ^Marco And it’s a balance. It’s a balance you have to strike. Obviously, if you’re hitting

⏹️ ▶️ ^Marco severe problems and severe scaling challenges, you’re going to have to go more in that direction of

⏹️ ▶️ ^Marco more complex code, and that’ll be more bugs. have things like weird caching

⏹️ ▶️ ^Marco bugs and things like replication delay or eventual consistency bugs depending

⏹️ ▶️ ^Marco on what kind of structure you’re using. And that’s all hard, that’s all complicated,

⏹️ ▶️ ^Marco and certainly you shouldn’t do that sooner than you have to in most cases. But

⏹️ ▶️ ^Marco again, it’s a balance. Just like you shouldn’t waste all your memory as much as possible when you’re writing a C program,

⏹️ ▶️ ^Marco or any program for that matter, you’ve got to find the balance. And with servers

⏹️ ▶️ ^Marco and scaling, I would lean a little more just because of the nature of you know, using someone else’s

⏹️ ▶️ ^Marco vast resources on their computer versus expensive shared server resources. I would lean

⏹️ ▶️ ^Marco a little bit more towards a little more complicated, but way more scalable.

⏹️ ▶️ ^Casey That makes sense. And far be it for me to argue with you, Mr. Tumblr.

⏹️ ▶️ ^Marco Thanks. Can I get that on a business card? All right. sponsors this week PDF pen for iPad

⏹️ ▶️ ^Marco new relic and igloo and we will see you next week

⏹️ ▶️ ^Casey now the show is over they didn’t even mean to begin because

⏹️ ▶️ ^Casey it was accidental accidental

⏹️ ▶️ ^Casey John didn’t do any research Marco and Casey wouldn’t let him because

⏹️ ▶️ ^Casey it was accidental. It was accidental.

⏹️ ▶️ ^Casey And you can find the show notes at

⏹️ ▶️ ^John ATP.FM. And if you’re into Twitter,

⏹️ ▶️ ^John you can follow them at

⏹️ ▶️ ^Marco C-A-S-E-Y-L-I-S-S. So that’s Casey Liss,

⏹️ ▶️ ^Marco A-R-C-O, A-R-M, A-N-T, Marco, Armin,

⏹️ ▶️ ^Marco S-I-R, A-C, USA, Syracuse. It’s accidental.

⏹️ ▶️ ^Marco They

⏹️ ▶️ ^John didn’t

⏹️ ▶️ ^Marco mean to. Accidental. Tech podcasts

⏹️ ▶️ ^Marco so long.

⏹️ ▶️ ^Casey I’d like to tell you about why I hate my car now.

⏹️ ▶️ ^Marco Wow. Is it because it’s, is it, did it get dirty finally?

⏹️ ▶️ ^Casey No, it is actually relatively dirty at the moment. Aaron and I decided

⏹️ ▶️ ^Casey this past Saturday to go to the local drive-in movie theater. And if you have a local

⏹️ ▶️ ^Casey drive-in movie theater, I cannot suggest enough that, assuming you don’t have small children,

⏹️ ▶️ ^Casey which I guess definitely eliminates Marco and probably eliminates John in this particular context,

⏹️ ▶️ ^Casey if you don’t have small kids, you can escape the kids, go to a drive-in movie. It’s really cool. And so the way

⏹️ ▶️ ^Casey the Gutchland Drive-In Theater works, which is between Charlottesville and Richmond,

⏹️ ▶️ ^Casey is you pull in, you pay something like $8 a person in your car, and you get a double

⏹️ ▶️ ^Casey feature. And you get the audio for the double feature by tuning to a FM

⏹️ ▶️ ^Casey station that the theater broadcasts. And it’s a really cool experience,

⏹️ ▶️ ^Casey especially on a really pretty night like this past Saturday was. And we have been several times in the past,

⏹️ ▶️ ^Casey although not for a year or two. And when we’ve gone in the past, we’ve taken my car, which at the time was

⏹️ ▶️ ^Casey my Subaru. What color was that? That was white. We’ve taken Aaron’s

⏹️ ▶️ ^Casey car, which was and is still a Mazda 6, which is a grayish silver,

⏹️ ▶️ ^Casey and we’ve never had an issue. Well, this past Saturday, I had just washed

⏹️ ▶️ ^Casey and waxed and leather conditioned and vacuumed Aaron’s car. And so I thought, well,

⏹️ ▶️ ^Casey my car’s dirty because yes, my car does get dirty. just takes longer than 40 nanoseconds,

⏹️ ▶️ ^Casey unlike Marco’s car. So we decided to take the BMW. The BMW has a push

⏹️ ▶️ ^Casey button starter, and it has an accessory mode. And so I figured,

⏹️ ▶️ ^Casey self, there will be no issue here. What we’ll do is we will go to the drive-in

⏹️ ▶️ ^Casey theater, I will put the car in accessory mode by turning it off and then pressing the push button once,

⏹️ ▶️ ^Casey and we will listen to the movie on the stereo, and it will be wonderful. I will turn off the iDrive display. I

⏹️ ▶️ ^Casey had to figure out how to do that because I completely forgot. And I will turn off the iDrive display

⏹️ ▶️ ^Casey and everything should be good and right in the world. So that’s what we did. And after 10

⏹️ ▶️ ^Casey minutes, I heard the BMW chime that you might remember from neutral. And

⏹️ ▶️ ^Casey it was telling me that my battery was dying. What? After 10 minutes of having

⏹️ ▶️ ^Casey the radio

⏹️ ▶️ ^John on. Doesn’t your car have two batteries? Or is that just Marcos’ car?

⏹️ ▶️ ^Casey I don’t think Marcos’ car does, does it?

⏹️ ▶️ ^John Not that I know of. car has like 10 batteries, right? I thought it only had one. You couldn’t you couldn’t find

⏹️ ▶️ ^John the one battery that one time. So we don’t

⏹️ ▶️ ^{John, Casey} know how many batteries are in this car.

⏹️ ▶️ ^John Who does this car could be filled with

⏹️ ▶️ ^{John, Casey} batteries?

⏹️ ▶️ ^Casey I thought they’re always on the passenger side in the trunk.

⏹️ ▶️ ^John No, wait, we went through this. Anyway, I thought his car had two batteries and I thought I thought that was a common BMW thing. But what do I know?

⏹️ ▶️ ^John I buy Hondas.

⏹️ ▶️ ^Casey It may be for all I know. But anyway, now it could be that I that the battery

⏹️ ▶️ ^Casey in my car is original. I believe it is and it was purchased originally in I think

⏹️ ▶️ ^Casey December 2010. So we’re getting to the point that maybe this is all a battery issue that I’m

⏹️ ▶️ ^Casey misconstruing to be a car issue. But what I can tell you is that the radio at

⏹️ ▶️ ^Casey least once turned itself off because it felt like it was tired of being on and

⏹️ ▶️ ^Casey it had thought that I’d left it on, not deliberately. And unlike in a keyed car

⏹️ ▶️ ^Casey where you physically put the key into accessory mode, in a push button car, it’s just

⏹️ ▶️ ^Casey being told, well, yeah, go put yourself an accessory and I guess we’ll hope that you don’t don’t turn yourself

⏹️ ▶️ ^Casey off. And it turned itself off. And then I turned it back on. And like I said, after like another 10 minutes or something like that,

⏹️ ▶️ ^Casey it started digging away about how the battery was dying. So we ended up watching only the first of the double feature

⏹️ ▶️ ^Casey by lowering the windows, which by the way, used a whole lot of juice because, you know, moving a motor

⏹️ ▶️ ^Casey is a heck of a lot harder than move than having an FM radio on. But I did that we lowered

⏹️ ▶️ ^Casey the windows and we listened through everyone else’s radios, because my damn car wouldn’t stay

⏹️ ▶️ ^Casey running and I was afraid I wouldn’t be able to crank it. And the other really interesting thing is even though

⏹️ ▶️ ^Casey they have little jump-starting boxes at the theater, I was so scared that

⏹️ ▶️ ^Casey it would get so dead and the jump-starting boxes would take so long to trickle-charge it that I wouldn’t

⏹️ ▶️ ^Casey even be able to push-start the car because it wouldn’t be able to engage the push-button ignition

⏹️ ▶️ ^Casey into run mode to get the thing so I could freaking push-start it. So it sort of ruined

⏹️ ▶️ ^Casey our entire movie-going experience. And granted, we’re talking about a sum total of $16, but it was really, really, really

⏹️ ▶️ ^Casey annoying. So now if we take my car to the movie theater again, if for no other reason than the auto

⏹️ ▶️ ^Casey disable, we’re gonna have to take a frickin’ stereo, like a boombox with us and keep

⏹️ ▶️ ^Casey it quiet so we can listen to the stupid movie.

⏹️ ▶️ ^John Yeah, it’s a good thing you already got her to marry you because you were not pressing your date on this night.

⏹️ ▶️ ^Casey It’s so true.

⏹️ ▶️ ^Marco Okay, two things. One, the fact that you can solve this problem by getting like a $20 boombox

⏹️ ▶️ ^Marco makes it a lot less of a problem. True. Two, do you think anybody

⏹️ ▶️ ^Marco who designed this whole like, you know, electronic push-button start and accessory mode and automatic

⏹️ ▶️ ^Marco turn off, do you think any of those people involved in those decisions have ever been to a drive-in

⏹️ ▶️ ^Marco movie theater?

⏹️ ▶️ ^Casey Oh, certainly not. But it’s still very annoying.

⏹️ ▶️ ^John You know, some of the recent iPods had FN tuners in them. Like the old iPod Nanos, you have those hanging around, but they

⏹️ ▶️ ^John have FN

⏹️ ▶️ ^{John, Casey} tuners.

⏹️ ▶️ ^Casey Oh yeah, we could do that with the headphone splitter actually. That would be totally ridiculous, but it could

⏹️ ▶️ ^Casey work.

⏹️ ▶️ ^John And also, you should just get a new battery for your car, because it’s ridiculous. And yes, batteries do go bad sometimes suddenly after

⏹️ ▶️ ^John many years. So you do have to replace them.

⏹️ ▶️ ^Casey Yeah, and actually, the more I think about it, the more I think that may be the issue in terms of the warning about,

⏹️ ▶️ ^Casey oh, god, there’s actually something that pops up on the iDrive. It’s increased battery discharge,

⏹️ ▶️ ^Casey because it gets very upset that the battery got as low as it did. And so the next time I take

⏹️ ▶️ ^Casey the car in, which will probably be for the N55 recall, that BMW just

⏹️ ▶️ ^Casey announced. I will certainly ask them to either replace it or double, triple check

⏹️ ▶️ ^Casey that it is as healthy as they claim it is. But man, what a stupid problem to have. I

⏹️ ▶️ ^Casey mean, maybe this is the biggest first world problem ever. It probably is. But

⏹️ ▶️ ^Casey what a frustrating, silly problem to have because an errands car, which has a key like most normal cars,

⏹️ ▶️ ^Casey none of this would have been an issue. And I also wonder if part of the problem was because the iDrive

⏹️ ▶️ ^Casey is a whole freaking computer, I wonder if the whole damn computer was turned

⏹️ ▶️ ^Casey on with a little 12 or 15 gig hard drive spinning and so on and so forth,

⏹️ ▶️ ^Casey even though the display was off. In other words, there’s not like a short circuit, if you will, just for

⏹️ ▶️ ^Casey the FM radio to keep that on and the whole freaking nav and everything was powered on just so I could listen

⏹️ ▶️ ^Casey to the radio.

⏹️ ▶️ ^Marco Do you think you could have brought the soundtrack to the movie on vinyl and maybe played it on a gramophone, that means

⏹️ ▶️ ^Marco you don’t need any power at all.

⏹️ ▶️ ^Casey Are we done here? Are we done? Are we done? I hate

⏹️ ▶️ ^Casey you.