Homo neandertalensis and Homo sapiens are similar in their brain-case morphology. The brain in the anatomically modern human (AMH) is globular in shape with large olfactory lobes, but otherwise Neanderthals resemble them. Study of internal organisation of the brain as an organ of course is nigh on impossible with fossils, so the extant primates have to be investigated to discover how the nervous organisation within the brain proceeds.
Eiluned Pearce, Chris Stringer, and R. I. M. Dunbar of Oxford University and the Natural History Museum performed this original task admirably and produce some really useful conclusions on the respective evolutions of our ancestors and their big playmates. Read it up in the Proceedings of the Royal Society B: Biological Sciences today.
First of all, the orbit size shows that Neanderthals had larger visual systems than AMH, alongside their body size differentials. Average orbits were 1404mm2 compared to 1223mm2 in AMH with commensurate endocranial volumes. This seems to compare with modern human and other species' measurements because orbital size tends to be an indicator of brain size. Neanderthals seem to have also had a higher lean body mass. The comparatively large brain compared to the body doesn't seem to have continued with the later Neanderthals, who didn't have brains any larger than their contemporary AMH.
A separate evolutionary path is indicated, perhaps involving different habitats or habits. The AMH brain size increase was in areas such as the parietal lobes. Their particular large brain size, in the same way as several extant primate species, is related to the size of their social groups and "mentalising competence." The novel conclusion the authors place on this is that," throughout human evolution, brain structure and cognitive function have placed a constraint on bonded group size and social complexity." Using this as a base, Neanderthals would have had smaller group sizes than their associated AMH.
Archaeology indicates that Eurasian AMH did indeed have larger, more geographically extensive social connections than their larger neighbours! If densities of populations were similar, the Neanderthals would occupy less area in Europe and Asia Resources, including trading capacity would be more restricted, leading to slower rates of innovation. Neural investment in our ancestors was in solving ecological challenges with relevant social adaptation, perhaps even in "having a bigger gang of neighbours or relatives!"
The Neanderthal alternative was to evolve larger bodies (and brains), maybe in response to cold, or just to remain as robust and resilient as the ancestral Homo heidelbergensis. Other species have found this approach effective against factors such as climate change in the higher latitudes in which these intriguing groups of hominims lived, loved and languished.