Is it separate from the tone change rules (tone sandhi)?
Yes it’s separate - this is not tone sandhi. Many compound words (i.e. words made up of 2 more characters) have a character change to a neutral tone. There are some common cases when it happens like most particles (的(de), 吧(ba), 嘛(ma) etc), common suffixes (i.e. 子 (zǐ), 们(mén) etc) when used as part of a compound word (e.g. 桌子(zhuōzi) etc) are almost always neutral tone. Duplicate words (i.e. 妈妈(māma) or 看看(kànkan) etc) also seem to have the 2nd character as neutral tone.
However for other words like your example it’s essentially random and you just have to memorize them. You’ll also find it’s regional - Taiwan seems to do it a lot less than mainland for the same words. Even with mainland China there is quite a bit of variation between regions as to which words have neutral 2nd characters and which don’t.
And to top it off it’s not only the last character of words - 差不多 (chàbuduō) has the middle character neutral!?!
Is it ever reflected in the pinyin?
Mostly. Every dictionary I’ve ever seen will show it correctly with neutral tones (although they may be region specific), but I’ve had mixed results with textbooks/graded readers - some just always have tones listed for everything even when dictionary says neutral tone on the 2nd character.
Where could I find more information explaining this phenomenon and why it occurs?
If you find a good source for why some are/aren’t let me know. Generally though it’s just that way because it is. Common particles/suffixes and duplicated words pretty much all doing it makes sense. But the 友 from 朋友 in your example? No idea why and never found any underlying logic to it - that’s just how the word is (or isn’t) pronounced.
Tone is sometimes not neutralized in reduplicated verbs 等等
Some nouns inexplicable have full tone on 子. For 男子, 女子 you could do theorize that the 子 is literal and through semantic drift the words now refer to adults. But what about 因子?
Sometimes the same native speaker will neutralize/not-neutralize the same syllable dependent on emphasis, tone, etc. I vaguely remember a stack exchange page where someone in a video pronounced 喜欢 the two different ways almost back to back.