Edge cities tend to spring up radially around larger population centers, leading to population distributions that tend to be proportional to the population in the host city and inversely proportional to the distance from it.
Decisions using the kNN framework are arrived at through a majority vote of an observation’s k nearest neighbors (given some distance metric). When aggregating many kNN decisions and weighing them against one much more important kNN decision, one strategy I’ve found to work well is to copy congress:
The critical neighbor is “the President” and can’t “pass” the vote, but can “veto” it.
A decision is made to “pass” either on the vote of a majority of the neighbors in the absence of a veto, or given a 2/3 majority in its presence.
One example of this is aggregating decisions over a market index. Each individual asset in the index has an impact in its overall movement, but the index itself (the President) can also be analyzed directly.
The root of this phenomenon is cultural, but in no small part it is perpetuated by the sheer number of hours that a job consumes. It crowds everything else out and both dichotomizes and quantizes the schedule: “weekday” vs. “weekend”, “work time” vs. “personal time”. Loving something that decimates one’s productive time so severely, or ascribing a disproportionately high level of importance to it, is an example of what Nietzsche termed “slave morality”. Doing so because one needs to fill time and has “nothing to do” when one isn’t working is even worse: it’s the behavior of the Last Man.
There’s little time to do anything beyond the job at that point. Choose carefully.
The fundamental dilemma of cartography is attempting to project a 3-dimensional structure (the Earth) onto a 2-dimensional map. This is mathematically impossible to accomplish losslessly, so distortions are invariably introduced. Several standard projections exist, each with its own advantages and disadvantages, but one thing I’ve never seen was a context-aware projection: one which maximizes distortion in the least significant parts of the map (based directly upon the data being mapped) in exchange for minimizing distortion in significant areas. Something like PCA.
Many spam mails that land in my inbox tend to be thematically similar, though the messages have slight variations (perhaps they’re being sent by the same spammer). Ordinary messages do not cluster so well. Clusters formed on these spam messages should thus be “tighter” than clusters to which ordinary messages belong. Cluster membership and validity may thus be used as a feature in subsequent spam classification.
The concept of original antigenic sin is something that can be simulated using data clustering.
The results of a long and interesting LinkedIn discussion:
We are conditioned to think of concepts such as “skill” and “talent” being specific to single fields and conferring little ability beyond the boundaries of those fields, when in actuality the boundaries are very arbitrarily drawn.
Children are all born polymathic. They must be, because there is simply so much in the world to learn: language, shape, quantity, motor function, reasoning, and causality among many other lessons. Even if brilliantly gifted at one of these, a child unable to use the rest would not thrive. The brain of a child is wired to learn as much – about as much – as possible, and for good reason.
Schooling supports this during the younger years. Primary education is modeled around establishing a foundational knowledge base in many areas. However, as one progresses further, the focus narrows, finally culminating in a borderline obsessive focus on a single narrow research question in a doctoral program.
Part of the problem at the root of this is the system of organic solidarity, in which everyone performs a single task and the pooled effort, coupled with a system of trade (money), keeps society functioning. Despite the negative aspects of this system, it is actually necessary for our current way of life, unless we wished to spend our days farming our own food. However, we do not have to take it to such a degree that everyone is expected to specialize in a single task. With the right combination of skills, individual workers would be more able to carry out both their own organic functions and those of others. Consequently, fewer workers would be required overall, less money would need to change hands, and the employers who hired broad workers would reap most of the profits. There’s an economic incentive; most simply have not yet realized it.
Another aspect of the problem, going back to education, is an emphasis on individual subjects rather than fostering general thought and creativity. Once the general reasoning ability is in place, it can be applied across many different subjects. I’ve thought of the intuition that comes with this as a more advanced level of “common sense”: because many areas of knowledge follow similar patterns, picking up on these patterns causes similar principles across many disciplines to become “obvious”.
However, because the reasoning taught in advanced education tends to end sharply at a disciplinary boundary (“cliff-shaped” rather than “hill-shaped”) without abstracting to general principles or crossing into other disciplines, the skills we acquire tend to also end at that boundary. This doesn’t indicate a deficiency or limitation of talent or ability, however – it merely represents the limits of one’s education and practice. However, it creates a fairly thorough image of talent or skill as a “single subject” phenomenon.
Even worse is the lack of a driving force for self-expression. I remember reading a study that surveyed the satisfaction levels of workers in demanding occupations, such as medicine and law. Those without a means of self-expression reported the least satisfaction of those in any occupation, while those with a means of self-expression reported the most satisfaction. In endless pursuit of answers that are correct or techniques that are proper, schools have left behind study of subjective, expressive, and creative problems, such as morality and autopsychoanalysis, and the result is a society that has forsaken humanity for a very narrow definition of competence, when in reality the two concepts are much broader and are in fact intertwined.
To resolve this final issue, I think we all need to go back to childhood in a sense… to the childhood sense of wonder and the desire to learn, create, and express. To make things not only because they are functional or profitable, but also because they are significant or bring joy to behold.
With sophisticated enough natural language processing, relationships between concepts and actions that can be performed with individual objects can be extracted and modeled as an ontology (sort of like object orienting the concepts in an email). Spam can then be classified based on how well the concepts fit into the ontology (unless it includes valid text, which some now does). An ontology trained on valid email would contain a wide array of concepts and actions that a spam mail would not possess, or would not follow.
For example, “Bob said” may be parsed into “Bob”, an entity, performing an action called “said”. “Bob enlarged”, however, is an action much more likely to be found in a spam email than in most people’s ham. The associations between concepts and actions can be used as features in spam classification.
Here’s a fun and revealing thought experiment: What would you do if stranded on a deserted island (no other people on it) with all of your basic needs (hunger, water, shelter, …) completely provided for as long as you remained?
Neat idea on the way home: wouldn’t it be nifty to “catch” pathogens as they infect the body by advertising “I am a cell ready for infection” (bet there are marker proteins for that), then perform the bait and switch on them as they make their move, walling them away and exposing them to the immune system as a form of proactive, adaptive, and needle-less vaccine?