Scientific Investing in Real World: Unifying the Extremes :Randomness, Determinism & Causality : Black Swan & Tail Risk Hedging

Scientific Investing in Real World: Unifying the Extremes :Randomness, Determinism & Causality : Black Swan & Tail Risk Hedging Black Swan, Tail Risk Hedging, Convexity , Causality, Dynamics in Financial Market Causal Adjusted Randomized Factor Investing Conceptual Book August 2023 This book is humbly dedicated to all the Investors & Of course our Mother Nature who originated thoughts in my brain to write this book for common benefit at large. (Note : Randomly written Deterministic Natural Thoughts in this Book in Randomized English Syntax rather than Conventional Completely Deterministic ones ! I’ll write about the Scientific/Mathematical Structure of Language e.g. English) itself ! Therefore, Please expect randomness in the way of writing e.g. repetition, typos ,a bit unorganized content, errors but that’s causally convex ! ) Author : Pankaj Mani Contact : (manipankaj9@gmail.com) About the Author : Pankaj Mani has written this book to share his own personal practical market trading, investing and risk experience and research views. He has personally engaged in interactions with many top-notch experts around the world like Nassim Taleb, Nobel Laureate Myron Scholes, Didier Sornette, Marcos Prado, Emanuel Derman, Espen Haug, Miquel Alonso, Raphael Douady, Paul Wilmott, Petter Kolm, Aswath Damodaran, Stephen Wolfram, Suman Banerjee, Arvind Mathur, Wayne Himelsein, Former Global Head-Goldman Sachs and many others across the world. He is a globally acknowledged research professional having been a trader, risk manager, and investor in the market having encountered tail events successfully like the recent Covid Crash as well. He has engaged in various acknowledged discussions/collaborations on various topics over the years with many world-renowned celebrated experts professionals, top fund managers, risk practitioners, and professors around the world. His scientific strategies based on the principle of convexity, causality, dynamics, and randomness management have done quite well over time in the real world. He is the President of the World’s largest quant finance community CQFI(Fitch Learning)- Delhi(India) Society, and also the Committee Member, the Global Association of Risk Professionals (GARP, USA) -Delhi(India) Chapter. He holds some of the world’s leading professional qualifications in finance e.g. CQF(Fitch Learning), GARP FRM, RWRI, AIFI@NYU Tandon, etc. He is Mathematician by Nature. Table of Content : Chapter 1 :Introduction Chapter 2 :Background of Modern Science Chapter 3 :Duality of Randomness & Predictability in Nature & Markets : Unifying the two extremes Chapter 4 :Understanding Randomness in Real World Randomness: Explained further in Mathematics : Riemann Hypothesis Chapter 5 :The Scientific Philosophy of Predictability & Uncertainty in Nature, Life and Financial Markets: Non-Stationarity as the Quantum Phenomena in Finance Chapter 6 :Human Nature Paradox Chapter 7 :Proof Why Forecasting Often Fails in Markets. Chapter 8 :Connection between Science ,Human Behaviour & Finance Chapter 9 :The Century Old Problem: Physical Brownian Motion(Einstein) & Bachelier Financial Brownian Motion Chapter 10 :Science of Energy behind Black Swan : How to deal with it in Real World Chapter 11 :Background Of Factor Investing Chapter 12 :Application of Statistical Tools in Finance : Misunderstanding the Role of Time dimensions in Real World. Incompatibility of Conventional Classical Mathematical/Statistical Tools in Finance Chapter 13 :Regression Analysis & Paradoxes Chapter 14 :Regression Part 2: Exogenous Condition Chapter 15 :Beta not Absolute but Relative Chapter 16 :Error Terms Hidden Dynamics Chapter 17 :Simpson’s Paradox : Real World Chapter 18 :P- Hacking , Reproducibility Crisis Chapter 19 :Paradox in LS Regression Methodology Chapter 20 :Simpson’s Paradox Revisited Chapter 21 :Beta as Time Dependent Parameter in Real World Chapter 22 :Simpson’s Correlation Paradox : Misleading Correlation in Real World Chapter 23 :Equity Risk Premium Puzzle Chapter 24 :Scientific Laws of Motion in Markets : Quantum Laws : Newtons’s Laws Principle of Least Action : Feynman’s Path Integral in Markets Chapter 25 :The Fundamental Issues with Conventional Value Method Chapter 26 :Redefining Value in Real Scientific Perspective for Finance Chapter 27 :Geometrical & Stability Equilibrium Approach to Fair Value Chapter 28 : How AI/ML could help Scientific Causality Further Chapter 29 :Causal Equation of Markets Chapter 30 :Scientific Background of Convexity & Randomized Approach in Finance : Quantum Energy Dynamics Connection Chapter 31 :Relativity in Convexity /Antifragility Chapter 32 :Complexity (Causality: Black Swan : Convexity : Scientific Causal Dynamics of Crash in Markets Chapter 33 :Causal Adjusted Randomized Factor Approach Chapter 34 :Understanding Risk in Real World Chapter 35 :Uncertainty Management is the Key: Causality Chapter 36 :Scientific Commentary on Markowitz Portfolio Theory : True or Not ! Chapter 37 :Revisiting the Benchmark as the Concept in Market. : Real World Alpha. Chapter 38 : Scientific Analysis of Tail Risk Hedging & Law of Risk Chapter 39 :Paradox of Risk : Law of Energy Chapter 40 : Tail Risk & Convex Strategies Chapter 41 :Key Result: Tail Event Paths Chapter 42 :Convex Equity Investment Portfolio Strategy Chapter 43 :Godel Results in Risk Management Chapter 44 :Statistics Vs Dynamistics : Causality Chapter 45 :Causal Theory Dynamics of Bubble and Crash Chapter 46 :Fundamental Issues in Conventional Probability Theory : Probability I Real World : Scientific Perspectives: Classical Mathematics to Quantum Human Behavioural Space-Time Mathematics Chapter 47 :Feynman’s Path Integral, Paul Dirac Principle of Least Action : Scientific Commentary on Backtesting, Causal Adjusted Monte Carlo Simulations Chapter 48 :Backtesting & Simulations in Causal World Chapter 49 :Possible Revision in Jensen’s Inequality Chapter 50 :Classical Mathematical/Statistical Tools to Development of New Quantum Human Behavioural Mathematics Chapter 51 :Current State of AI/ML in Finance: Fundamental Issues : Non Stationarity as the Quantum Reflection of Human Behavioural Randomness Chapter 52 :Summary 1.Introduction : In this Randomly written yet Deterministic thought practical book I’ll try to humbly unify the two greatest extremes of Randomnes and Determinism through Scientific Causality : Predictability and Uncertainty : Black Swan and Tail Risk Hedging that humankind faces especially in financial investment and risk domains practically that involves trillions of dollars of investments globally affecting the lives of fellow common people/investors at large directly or indirectly. I discuss about the Scientific Perspectives of Financial Investment approach e.g. Factor Investing. First will talk about Philosophy & Development in Science and its connection with Financial Markets. How Investment/Factor Investing can be made scientifically forward looking unlike conventionally backward looking . Will talk about the Causal Laws of Motion in Markets and how that should be applied to Factor based investing . How Value as the factor should be redefined scientifically in Real World Investing unlike traditional understanding of Cheapness and how that has also been linked to flawed conventional statistical infrastructure. Will discuss about some fundamental misconceptions of popular statistical tools e.g. Linear Regression Analysis, Backtesting etc. Will also discuss about the fundamental issues of compatibility with the application of Classical world mathematics in finance and switching to Quantum world mathematics as the future development. The Role of Time & Relativity Dimension in Statistics /Mathematics in Real World finance. Will discuss about the Scientific Origin of Black Swan, Convexity and finally a New Causal adjusted Randomized Factor based Investment. Will also talk about Scientific Perspective behind Tail Risk Hedging, Paradox of Risk from Scientific Perspective practically. Will also check the scientific validity of Markowitz Portfolio theory, CAPM, historical mystery of Equity Risk Premium. Will talk about General Theory of Relativity in Financial Space-Time. Will discuss about Tail Risk Hedging Strategies, Convex Portfolio Building, Anti-fragility all from practical scientific perspective Will also discuss that ultimate intellectual maturity is the acknowledgement of duality/two extremes in Nature in the form of Randomness (Uncertainty) & Determinism(Predictability) like Wave Particle Duality. Rather than being competitor to opposite being complementary to each other. It attempts to unify the two extremes of Black Swan Randomness and Causality in the dual form. In a nutshell the book humbly explores possibly new practical way of looking at finance scientifically in forward direction of time based on my humble experience in markets and having engaged in discussions globally with some of the top brains and how that should be implemented to revolutionize traditional investment theories and affecting the lives of common investors at large. Note : I’m not going to make it much technical in mathematics rather conceptual behind mathematics, for to be understood by all. Mathematics itself needs fundamental transformation and evolution here for application ! Also, there exist fundamental issues with the conventional statistical and mathematical tools, which is blindly applied in finance without understanding the concept. So, my focus is on concepts behind the mathematical/statistical tools, which are often misused and misunderstood in finance. Also, there are some fundamental compatibility issues in the existing classical world mathematical/statistical tools applied especially in finance driven by quantum human behaviour . Hence applying the same faulty mathematical tools doesn’t make sense. I mean that traditional mathematical tool used so far are not well capable of describing these natural aspects. In future they will possibly have to be developed new mathematics for quantum human behavior because current existing tools like statistics and calculus etc . are not fundamentally compatible to deal with the quantum human behaviour which drives the markets. Hence in this paper, applying these existing conventional mathematical & statistical tools don’t make much sense ! Also, there is a Self-referential problem like in the famous Godel Incompleteness Results(in Mathematical Logic) in this paper that disclosing the things beyond a limit could influence the observers(trader/investor)’ minds who could be themselves the part of markets. So, in this way, Causal dynamics of Quantum market could be influenced. Hence, there would remain a limit on how much to disclose publicly. P.S. Kindly ignore typos in grammars, spelling,repitition etc. as even they are random to varied extent for me relatively! Please take this paper as ongoing rough work. I know some would agree, some wouldn’t on certain points, so need not worry. Scientifically we live in many worlds in Quantum perspectives, So, there couldn’t be absolute existence of the perspective fit for all rather contextually true. But yes always open and humble to discuss and improve collectively. Random Collection of Deterministic Thoughts Here, we talk about some of the most foundational issues that have been at the core of financial quant investment and risk industry over many decades since beginning affecting the common investors, institutions at large globally managing huge money in trillions of dollars. So, I’ll cover the following fundamental issues succinctly. Scientific Mechanism & Philosophy of the Universe and Nature : Philosophy & Principles of Modern Science & Causality : Newton to Feynman Science of Financial Investment & Risk Management Foundational issues in the traditionally popular approaches like Factor Investing. Unify the two extreme approaches in the investment industry and of course nature i.e. Predictability & Unpredictability (Randomness) . Godel Incompleteness Theorem'(One of the most important discoveries in mathematics and logic ) Relevance in Mathematics, Science & Implications for the Ultimate TOE( Theory of Everything in Science & Finance. ) Unification of Causality & Black Swan as the Complementary approach. Understanding Randomness Scientifically as Causality Foundation Philosophy and Mechanism of Conventional Statistical tools like Linear Regression etc. Fundamental Loopholes. Foundational Concerns of Conventional Probability Theory in Real-World Finance. Classical Space Mathematicas to Quantum World Human Space-Time Mathematics! Mystery of Simpson’s Paradox : Hidden Information about Randomness & Determinism Role of Time Dimension in Mathematics & Statistics in the Real World & its foundational implications in the finance and risk. Fundamental issues with Backtesting etc. Foundational issues with Conventional Probability Theory :Causal Adjusted Real World Probability Causal Based Monte Carlo Simulations in Real World Finance & Human Space-Time from Scientific Perspectives A Revisit to the Connection between Classical Laws & Quantum Laws Laws of Human Behaviors in Quantum world : Connection with the Laws of Nature & Finance. Causal Mechanism which drives the markets scientifically. Scientific Equations of Laws of Motions of Stocks Fundamental Issues in Geometric Brownian Motion in Finance. Causal Version Economic/ Financial Space-Time analogous to Einstein’s General Theory of Relativity Valuation in the Real World. Valuation as the Scientific Concept: Misunderstandings & Miscalculation of Valuation in Conventional understanding of Factor Based Investment Theories. Understanding Market Scientifically from Laws of Human behavior coming from the Laws of Nature. Understanding Randomness in detail scientifically. Century Old Problem of Financial Brownian Motion by Louis Bachelier & Physical Brownian motion applied by Albert Einstein : The Commentary and Latest Findings. The Mystery of Historical Equity Risk Premium Puzzle in Finance : Explanation of the Mystery. Role of Randomness in Nature & Markets and the Origin of Black Swan Type Events Managing & Strategizing Randomness in the Real world. RCT type algorithms like Algo wheel experiments in real-world markets. Complementary to Causality .The Scientific Link to the Nature’s mechanism Randomized Factor Approach to Portfolio Management How Risk is often miscalculated and misunderstood conventionally? How Risk should be measured in the Real World ? Law of Risk ,Paradox of Risk, Scientific Crosscheck of Markowitz CAPM Risk-Return Perspective and New Scientific Perspective Scientific perspective of Tail Risk Hedging. Fundamental Issues with Conventional Machine Learning Tools The Way forward 2.Background of Modern Science : Modern Science has evolved from Newton’s Classical Laws to Quantum Mechanics over the time. Einstein’s Theory of Relativity to Schrodinger’s wave Equation to Heisenberg Uncertainty Principle to Paul Dirac’s Principle of Least Action to Feynman Path Integral Approach have been developed in the study of science over the time. Science is still trying to unify the two extremes of Classical & Quantum world laws. On one hand, things appear Deterministic in day to day Classical world, things appear highly Uncertain/Random/Probabilistic in Quantum world. Modern Science is trying to understand the Quantum world of Uncertainty and Unifying it to the Classical world. Albert Einstein throughout his life couldn’t digest the belief that God is playing dice with us. He believed in Deterministic world inherently which has been refuted over the time. On contrary, Principle of Complementarity emphasizes the role of Observer(Human Brain) through which the External world is experienced. Even Stephen Hawking talked in context of theory of Everything in Science about the same regarding the role of human brain in the ultimate theory of universe.. Max Planck ask talked on the similar line. This is infact true that Science can’t find the ultimate theory of Everything without knowing about Human Brains, Consciousness etc. There is also Paradox of Self Consciousness and I wish I could take this here to show how Modern Science needs radical approach and try to unify the ultimate reality of the Universe- Brain mutually. How a peculiar super geometry could unify the Copenhagen/Quantum and Classical Interpretation of the universe where the external world exists independent or dependent on the observer. But that’s extremely deep and require outstanding imagination but as this is beyond the scope of this paper, I would leave it here for now for some science paper/book. This is beyond the scope of this paper here. Hence, I would like to confine here. Feynman’s Path Integral approach has been arguably the most powerful scientific result in Science which originated from Paul Dirac’s Principle of Least Action. I would state based on my own scientific research that even Classical world theories like Newton’s Laws of Motion etc. originate from the Quantum Energy Laws fundamentally. In context of deeper aspets of Zeno’s paradox , I would say that Classical world laws are just the superficially an illusion created by Quantum Realities. But that’s deep subject for some other paper. Those who have studied Physics know that one can derive Newton’s Laws of Motion from Principal of Least Action which is also supposed to be the core of Quantum world Laws. It states that any system tries to trace the path of Least Action (Action is basically a function of Kinetic & Potential energy),loosely speaking, least energy in least time. So, Feynman’s Path Integral approach which also the principle behind Feynman ;Kac approach for solving Differential Equation having use in Quantitative Finance originated when Richard Feynman tries to show that Schrodinger Wave Equation in Quantum Physics can be derived from Feynman’s Path Integral approach. Feynman’s Path Integral tries to sum all the possibilities of trajectory to find out the resultant path and this came up when he tried to explain wave particle duality in context of the famous Double Slit Interference experiment.. Though this approach by Feynman needs further improvement and clarifications in my own ongoing research especially in Human Quantum Behavioural context , that’s beyond the scope of this paper now but that’s the foundational guide to how financial market tools must be redeveloped scientifically in the real world future applications replacing the ineffective old ones. Infact Especially the Quantum Human Behavioural world, as Feynman calculates the Causal Probability of different Possible Paths and then sum all of them, In Reality the Paul Dirac’s Path of Least Action may not be the Sum of All Paths rather the Path of Maximum Stability & Likelihood. This is where the fundamental revision requires in Quantum Human aspects driving the financial markets based on my research . Quantum Human Version of Feynman’s Path Integral /Paul Dirac’s Principle of Least Action. Maximum Stability Path may not be the Sum/Average of All Probabilistic Paths rather always in especially Quantum Human Behavioural Driven world which drives the market. This is also linked to how Conventional Probability & Statistical Expectation Value need to be revised in Market Context Causally driven by Quantum Human World.I’ll talk about it later and show how the powerful tools like MC Simulations, Statistical Expectations & Statistical Tools have to be made Causally Scientific in Real World Fiannce! Feynman’s path integrals formula is so far possibly the most powerful theory in physics to describe the laws of Nature. Starting from the Paul Dirac’s theory that a body traces the path of least action(roughly speaking least energy and time ), Richard Feynman derived Path Integral approach to discover the path of an object. It basically extends Principle of Least Action to Quantum world from Classical world. Below are the equations of Feynman’s Path Integral theory to Principle of Least Action which tries to minimize the Energy Function of a system from which are linked to the Laws of Motions Principle of Least action : \mathbit{E}_\mathbit{k} is Kinetic Energy while E_{p\ }\mathbit{is}\ \mathbit{Potential}\ \mathbit{Energy}\ \mathbit{of}\ \mathbit{the}\ \mathbit{system}. Still Science has lot to deeply look into like Locality-at-distance by John Bell(Bell’s theorem ,Quantum Entanglement, Superposition, Interconnectedness, EPR Paradox (this year Nobel was awarded on the same Bell’s theorem related work in Physics ) for some wonderful experiments. I could humbly try to take Science to much deeper level to explain my own findings here but that I would deal with separately the mystery of Locality-at-distance which Modern Science has not probably imagined so far and also Modern Science needs to push it’s boundary beyond and obviously the Crucial role of Human Consciousness comes into play. Infact it’s the super geometry at play behind the Quantum Entanglement & Universal Interconnectedness unifying the Consciousness of the Observer and the External Physical Realities. Will talk about in some other paper. Sorry I get drifted to talk more deeply about science here ! But that guides how serious finance should be done in real world in causal scientific approach. But controlling myself here, what is relevant in Finance context that Market and hence the dual game play of Uncertainty & Predictability would keep on going endlessly until investors. Market is essentially the sum of Quantum Human Behaviours of Buyers and Sellers in the Market and Stakeholders etc. Even Feynman’s Path Integral approach/ Paul Dirac’s Principle of Least action follows for Human Quantum Behaviour in Markets but yes slightly different variable form rather constant as in non-living world physics. Hence, while Market/Finance has to be studied Scientifically in Causal way, one has to acknowledge that Science won’t do magic by being able to Predict the Fundamental Uncertainty inherent Everywhere in Nature including Human Behavior. But there is a good news to deal with Uncertainty which I would cover in different section here on Randomness. Infact the whole philosophy of Prediction using the Deterministic Mathematical /Statistical Tools seems to be misguided scientifically. Prediction tries to believe in disrespecting Quantum Uncertainty in Nature, Human Life etc. That’s doesn’t mean I am advocating against Prediction rather acknowledge the scientific Fundamental role of randomness, Uncertainty in Nature in Complementary way. And scientifically, the best way of Prediction is not to Predict beyond a point and respect Uncertainty in Nature that exists as Wave Particle Duality type. I’ll explain that in detail in my book. Infact, Conventional Mathematics itself is Deterministic say Conventional Probability Theory, Statistical Tools are themselves Deterministic!! Beyond that as traditionally, statistical tools e.g. Backtesting etc. are applied in finance are mostly backward static looking but by looking it into science like dynamical motion of stock in Quantum Human Behavior Space-Time, trajectories would be forward looking like hard science Physics applied to Human Context. It would help understand and manage Black Swan type events scientifically and better be prepared to manage portfolio for people at large which affects common innocent people’s hard earned money at large including through pension, Sovereign Funds , Mutual funds, hedge funds etc. Quantum Zeno Paradox /Turing’s paradox in Markets. One of the most important quantum phenomena is Quantum Zeno Paradox also known as Turing’s paradox. Classical Zeno' Paradox pointed out that motion is impossible . Quantum Zeno paradox is that more frequently an observer observers, it arrests the change in the time evolution of quantum system . I’m not going deeply into that here but the contextual point is that market being fundamentally a quantum system, there is Quantum observer’s influence as well. The more frequently a trader/ investor tries to observe at market, there would be more arrest of time change of the evolution of the system. This frequent time to time observance by traders /investor could delay its time evolution in some way affecting the growth over the time, consequently the risk- return. Hence to make money in the quantum market, trader has to allow the optimal time frame without influencing the quantum evolution of the portfolio. This is a very deep topic and requires proprietary strategies ! Hence I’ll stop as it would likely affect the Causality of the Markets. Yes that’s true. Causality can be affected to some extent by writing about Causality publicly !! Self-referential issue !! 3.Randomness (Unpredictability, Uncertainty)Vs Determinism(Predictability): Duality in Nature.: Unifying the two extremes For Long there has been serious conflict going on in Science/Philosophy/Social/Financial domains whether the world is Random or Deterministic. Science is still far from solving this. But let me state that there exists Fundamental duality in the Universe in terms of Randomness & Determinism just like Light behaves as Wave or Particle , which is fundamental duality inherent in the Universe. Similarly , there exists Fundamental duality of Uncertainty & Determinism in Nature and that’s well reflected everywhere including Financial Markets. Market essentially also has this duality. And this exists Relatively to the Observer. Same thing can be Random or Deterministic to different Observer relatively. Information availability is also one of the causes. Also, same thing can be Random and Deterministic both to the same observer in different perspectives. This is Paradox of Randomness. Infact Randomness & Determinism are NOT absolute phenomena . They can both co-exist to the same observer from two different perspectives and also relatively for different observers. It just depends on perspectives and also level of information. Paradox of Randomness can be resolved by understanding this Fundamental duality in Nature, hence markets,life that Both co-exist together perspective wise.. It is generally treated that there is Absolute Randomness existing, that perspective needs to be Fundamentally changed ! It’s Relative not just for different Observer but also for the same Observer!! One and to understand this Duality inherent in Nature and everywhere in Life, markets. And I must say it is fundamentally linked to Wave-Particle Duality in Nature ! That’s deep and beyond the scope of this paper. At the same time, I must categorically mention that Causation & Random are not two opposites rather even Randomness has the Cause. But knowing that something is Causal doesn’t mean it’s completely deterministic. It all depends on the level of information. Like we know the Causal Law of Motion of a Car but that doesn’t mean one can completely predict an accident !! Randomness would also be deterministic for itself. Reference frame matters. So, Randomness is NOT Randomly Random but “Deterministically” Random. ! Things are Locally Random but Globally Deterministic in Nature, Universe and even Markets. One can relate this to Quantum Physics where a particle behave randomly at individual level but highly deterministic at collective level. But again Deterministic doesn’t mean Completely Predictive ! Hence, contrary to the understanding that Randomness is opposite or different from Causality and Causality is overestimated by Humans, I would rather say Randomness does have Causality and even Randomness has inherent Determinism that need to be discovered but that doesn’t mean they would be completely predictable. It’s like Quantum world as explained earlier, Fundamental duality exists. Things could be random and predictable relatively. As in Quantum world, a wave could behave Random/ Uncertain at some level but Predictable as well at some other level. So, Randomness has to be studied deeply as the part of Causality but one has to acknowledge its existence and uncertainty inherent in Nature Relatively and Things can’t be completely Deterministic or Completely Random. Randomness doesn’t mean anything can happen, it would always be driven by Causal forces ,but that we may not completely know. One an relate this to Complexity, Non-Linearity etc. Hence, Randomness is Deterministically Random not Truly Random in Nature/Market/Life hence in Financial Markets as well. As usually understood , Randomness is not absolute phenomenon, same thing can be random and deterministic from different perspectives and availability of information relatively for the same observer and different observers too. It’s like Duality. Non-Stationarity in Financial Data comes out of this Randomness originally.. Randomness does converge to its Equilibrium State ! This is governed by the Law of Energy and Nature.This is Deterministic Law of Randomness/Uncertainty!! Yes Looks Paradoxical but true in Nature ! So, things – On needs to understand the Causality /Determinism behind the Randomness. Randomness does follow some Deterministic aspects but that doesn’t mean it would be completely predictable. That’s the key to success in real world It’s linked to attaining more and more Convexity. Will be dealt in the later section. 4)Understanding Randomness in the Real World : Randomness in Mathematics :Riemann Hypothesis We often talk about Randomness in Nature, Markets, Life etc. but seems like we have to understand them deeply. Before we explore Randomness in finance, let’s first take ourselves to Science. As it is often said: Life is unpredictable, and random things happen to us all the time. But yes they are also predictable in some perspectives. One might say the universe itself is random. Yet somehow, large numbers of random events can generate large-scale patterns that science can predict accurately. Heat diffusion and Brownian motion are just two examples. Recently, randomness has even made the news: Apparently there’s hidden order in random surfaces, and we may be close to seeing a quantum computer generate ultimate randomness. This latter quest for perfect randomness is important because randomness brings unpredictability, and all non-quantum attempts to achieve it have the hidden flaw of being generated by algorithmic methods which can, theoretically, be deciphered. . Is nature inherently random? According to some interpretations of quantum mechanics, it is, explaining why we can’t precisely predict the motions of single particles. In the famous double-slit experiment (which, as Richard Feynman declared, “has in it the heart of quantum mechanics”), we cannot predict where exactly an individual photon passing through two slits will land on the photo-sensitive wall on the other side. But we can make extremely precise predictions of the distribution of multiple particles, suggesting that nature may be deterministic after all. Indeed, we can predict to many decimal places what the distribution of billions of photons shot at the double slit will look like. This dichotomy between unpredictable individual behavior and precise group behavior is not unique to quantum mechanics. There are many novel and strange aspects of quantum physics — particle-wave duality, quantum entanglement and the uncertainty principle, for instance — but probabilistic equations that give precise predictions of ensemble behavior are not among them. We see this phenomenon wherever very large numbers of like elements interact, such as in thermodynamics, where we can predict collective measures like heat and pressure with precision, though we may be completely ignorant about the paths taken by individual molecules. There is a debate whether randomness or determinism lies at the heart of quantum mechanics, which I characterized as team B (Niels Bohr) versus team E (Albert Einstein). Team B sees the unpredictability of particle behavior as evidence that at the fundamental level of the universe, determinism is replaced by intrinsic, objective randomness. Team E contends that this randomness is merely a sign of our ignorance of a deeper level of deterministic causation. The great mathematician and one of the founders of algorithms and computer science, John Von Neumann quoted that any algebraic system producing truly random numbers in mathematics is just ignorance for they are the output of deterministic algorithm. Let me put my scientific point of view because that’s where the origin of Fooled by Randomness, Black Swan etc. originate in Finance and Market. Fooled by Randomness essentially seems to believe in the Niel Bohr school of thoughts of pure randomness when he talks about the Role of Randomness in Life & Market ? But it may also mean that even if Randomness doesn’t exist truly , it exists due to ignorance of the observer Relatively. That means by analysing Causality inside Randomness, one can understand it deeply. This point may be open to debate. Although he is right in many aspects that Randomness does exist in Nature ,Life, Markets, but it’s not Randomly Random rather Deterministically Random. I would categorically state that Randomness is Deterministically Random not Randomly Random. What I mean here is that Even Randomness has Causality and they follow some Deterministic Laws of Nature.. Things appear Random Locally but they are highly Deterministic Globally in Nature. One can even visualize this in Mathematics as well. Riemann Hypothesis is termed as one of the most important problem in Mathematics which is linked to the Prime Number Distribution. The existence of Prime Number appears Random Locally like Riemann Zeta function’s Non-Trivial Zeros but they are highly Deterministic that Non-Trivial Zeros linekd to Prime Number lie on Critical Line. This has been experimentally verified as well which I think it’s true. Riemann Hypothesis Pictures Similarly one can see at many places in Mathematics as well. One has to fundamentally understand that Nature is work in progress. Randomness is the part of evolution of Nature locally like wave but they are highly Deterministic Globally like Particle.. Infact, Brownian Motion is also fundamentally/mathematically linked to Riemann Zeta fuction. Hence I had written a paper earlier to show that based on the Riemann Zeta Link /Riemann Hypothesis, even Brownian motion could be fundamentally globally Deterministic even if Random Process locally ! I had also devised an approach many years ago to show Riemann Hypothesis would be True by imagining the whole mathematics as a physical system but that’s an open problem as of now officially ). Based on that If they truly describe markets, Markets are also globally Deterministic, despite looking Random. This could also raise Fundamental question on the Brownian Motion/Stochastic Equations as the infrastructure to describe Randomness in Markets which was first done by Louis Bachelier, if market is really random !! Collective dynamics like Quantum world is that Summation of Randomness Leads to Deterministic patterns.. That’s how Nature Fundamentally evolves. That’s also the reason why Monte Carlo Simulations type algorithms might work . Also, RCT, Algo Wheel type Randomized approaches.. There exist Global Causality of Local Randomness ! Randomness does follow the Law of Nature & Energy Deterministically in the Universe hence Human Behavior, Markets… That doesn’t mean things would be completely predictable. It’s like we know the Laws of motion of a Car but can’t exactly predict an accident ! The dynamics of Randomness comes from the Origin of Principle of Least Action by Paul Dirac to Feynman Path Integral…Things follow this path to reach the most equilibrium stage…And Randomness too essentially follows that Law ! That’s why even Portfolio Construction should be based on Randomized approaches like Algo wheel where Factors can be Randomized and the Best Path of Least Action could be selected……This is what essentially leads to Convexity approach scientifically. Hence , Black Swan ,Randomness do exist but essentially they would follow the Law of Causality where they would revert to their equilibrium state. The key to Risk Management is to survive the whole cycle before the convergence !! Randomness in Nature is not Randomly Random but they are Deterministically Random ! Randomness does follow Deterministic Laws of Convergence to the Stability being the form of Energy. Deterministically Randomness has well defined Causal Mechanism but that doesn’t mean we can completely predict them ! It means they have duality . They are basically Random but even inside Randomness things have certain causes and are governed by the causal laws ! Random is Relative term depending upon the level of information with the Observer. So, we may not know what exactly would happen but that doesn’t mean anything can happen anytime. What ever would happen is governed by the causal laws but yes even then we can’t predict them completely due to Incomplete information availability. Hence we often come across Randomness & Determinism. But in Real World we have to define the term as “ Deterministically Random” & “ Randomly Random if so. Even Randomness follows the Laws of Nature and has proper Causality (But it doesn’t mean completely deterministic !)but yes there would likely always exist incompleteness of information leading to Randomness relatively for a human observer at any moment in time. But,I humbly believe there needs to understand in depth the Causality(Deterministic aspects) behind Randomness and its Laws to understand it more scientifically. It would help more in achieving the higher Convexity.” “We have to technically understand that Randomness in Nature hence Market & Human Behavior is NOT “Randomly” Random rather they are “Deterministically” Random in Nature and Markets ! Means Randomness doesn’t mean anything could happen randomly. Whatever happens (we might not predict completely due to lack of complete information available at any time )as the fundamental principles inherent in Nature inherited to Markets but it has certain causal determinism Fundamentally!” – Randomness has Hidden Synchronicity ! Things have inherent Randomness locally but Globally they are bound to be Quite Deterministic ! But Determinism doesn’t mean we would be able to predict completely ! There would always be incomplete information of forces and their Measurements to be able to make complete Prediction in the Causal System. So despite knowing the Causal, we might not Predict completely and there would be Randomness. In a nutshell, Even Randomness has deep Causal Mechanism in Nature and hence markets as well. even Causal Mechanism has Randomness vice versa .It’s deep paradox in itself scientifically ! But that’s how Nature works and hence the markets as the subset of that. Randomness exists because the Nature is itself a Work- in-Progress. It’s not Complete in its own Construction ! Nature itself might not know where Nature could be as it’s itself under Construction. But that it does have Causality and would depend on how various forces work out. Further, Relativity of Randomness & Determinism - What could be Random for Observer X could be Deterministic for Observer B. It’s Relative ! Hence proponents of opposite schools of thought have to understand this “Relativity” rather claiming to prove them “absolutely”. Also, something could be Random and Deterministic both for the same observer in different perspectives . The deeper point : like Wave-Particle Duality, Randomness and Determinism co-exist as duality !! Nature, Human Behavior & Markets are not Randomly Random but Deterministically Random !! Randomness does have hidden Causal Mechanism but we won’t be able to completely predict the reason being we don’t know different forces and their Measurements like Newton’s Laws of Motion . We know the Cause but we don’t know the complete information about the measurement of different forces in the system at any moment in time leading to Unpredictability. Size, we have to understand the Scientific reason behind Unpredictability & Randomness. They don’t exist absolutely rather relatively. 5)The Philosophy Behind Prediction : Trying to make the world Deterministic and Banishing Fundamental Uncertainty from Nature, Life & Markets. Non-Stationarity as the Quantum Phenomenton in Finance Almost all the financial models try to predict the market based on various statistical and mathematical tools. Even in Life, we want to predict and also in various other social and other domains. Everyone talks about Prediction more or less. But have we ever thought deeply that what we inherently do when we try to predict ? So, what does Prediction mean scientifically? Prediction means the observer is trying to make the world deterministic by knowing the state of the world at Time t at present (Time =0). Even Nature doesn’t know what it could be at Time T =t because it’s itself a work-in- progress driven by Causal forces. And the State of the world at Time T=0 and the Time T=T both could be different and even Nature doesn’t know what could be. So, by trying to predict, the observer might influence the system by influencing itself. These all are Scientific in context of Quantum Reality in Physics. But is Universe Really Deterministic Completely ? Is it not against the Quantum Law of Nature ? Let’s imagine the world is completely Deterministic or Random ! Will the world run ? No, it will cease to run eventually. The only way the Universe, Markets, Nature,Life could Function is the existence of Duality & Opposites. So, by Trying to Predict more and more, we are trying to make the system more and more Deterministic which it’s Fundamentally not. Infact there is paradoxical underlying truth . So, having origin in the Quantum world science, an observer should not focus much on Predictions because by doing that more beyond a point, his own predictions could Influence his actions and that would likely change the Prediction only. In scientific terms, by prediction means trying to make the world Deterministic, things would become more uncertain. And the more one focuses on managing Uncertainty and less on Prediction beyond a point, the more Deterministically the result would occur. So, this Paradoxical situation originates due to Quantum behavior. Even Market is a Quantum wave coming Human Behavior etc..So is life. Hence, to get better result, one must admit and respect the Duality in Nature, and not try to make the system moreand more deterministic by trying to predicting more and more endlessly. So, the focus on process and present Uncertainty over the time would automatically lead to great future even without knowing the future in advance deterministically. On the contrary by focussing more than enough on future prediction, the present is often causally affected which eventually influences the prediciton/future. This is no non-scientific thing rather very much scientific one coming from Quantum world laws of Uncertainty, Observer’s influence on the outcome etc... In other words, one has to understand technically there would be lot of causal forces between now and the time for which it is predicted..and that could change the Prediction Causally. It’s like observer is trying to predict the motion of a Car without knowing the causal Forces thoroughly as done in say Newton’s Laws of motion. This being a dynamical causal system like a physical system, there has to be understood the causal forces being applied. So, how can one predict scientifically beyond a limit without knowing the causal forces and their strengths etc. It would be just a guesswork then. I must repeat that Quant models must focus to manage this inherent Uncertainty in Nature which is quite Scientific and yes it inherits from Quantum Uncertainty to Human, Markets & Life etc. Although it may be hard to imagine for some, but it’s scientifically like Quantum effect in Markets and Life as well !! The more an observer tries to predict, the more quantum uncertainty it would get. The less he tries to predict, more certainty would the things become. By predicting more and more beyond a point, the observer tends to change its own predictions using quantum effect. As in Quantum physics, Observer’s role is critical and could influence the outcome. Recall Heisenberg Uncertainty, to Schrodinger’s Cat to Quantum Entanglement in Quantum Physics.So, Observers have to understand the critical point and maintain the balance between Predictability & Unpredictability ( Uncertainty) inherent everywhere ! Hence, it’s scientific based on quantum uncertainty effect that the best way to predict is not to predict it beyond a point. It’s better to focus on the local process in a convex way, automatically, the outcome would get accomplished. The more certain and predictable observer/trader/investor would try to make the outcome by predicting more and more beyond a point , the more it could be unpredictable . This is Quantum World Effect in Market , Non-Stationarity in Finance as the Quantum Reflection of Human Behavior :An Opportunity rather Constraint !! Non-Stationarity of Financial Data in Real World, a fundamental issue with almost all the Financial Models is basically the reflection of Quantum Human Behavioural world that is being mapped onto classical mathematical/statistical models to predict etc.. What needs to be conceptually understood that no matter how sophisticated models based on stationary mathematical tools(e.g. classical tools) are developed,the prediction will be constrained by Non-Stationarity being a Quantum Reflection of the Reality ! The Conventional Classical Mathematical Tools are fundamentally Stationary But practically here, Non-Stationarity can’t be resolved by developing advanced models as long as their mathematical foundation is itself stationary !! Even the Fundamental Philosophy of Prediction will also always be constrained by Non-Stationarity. So, the key is not to develop complicated stationary models rather understand the causality dynamics of quantum human behavioural randomness from where non-stationarity enters in the financial time series data. As discussed in the Scientific aspect of Randomness in this paper, it can be possibly resolved. In this way, Non-Stationarity should be like an opportunity rather a fundamental constraint here ! In simple way, to predict like Newton’s Laws of Motion in Physics, one needs to have the information about all the causal forces. But does one have this, No ? Infact there is a Self-referential issue as in the famous Godel Incompleteness Theorems. In Science or in Markets, the fundamental issue is that Observer tries to Predict the system,who is itself the part of the system. The System is not independent of him/her. So, scientifically, Predicting the Market/Nature/Universe/Life is Causally Predicting the Self ! Now that’s Contradictory! Can we predict ourselves ? How far ? Market is essentially the summation of Human Behavioral Forces. If an observer can’t completely Predict itself, how come it can predict the Resultant Sum of All the Humans completely? Infact this is the fundamental challenge of Science as well where it’s the concern that without understanding the Human Observer Consciousness etc, will there ever be the Ultimate Theory of Everything for the Universe?? No . Caution : I don’t mean own should not predict at all or the world is is not at all predictable. Because, if Everything becomes Unpredictable/ Random, that would also cease the system to run ! The point is observer needs to appreciate and admire the existence of both the sides and act optimally. So, the key is not to try to predict more and more beyond a point rather act in a balanced way appreciating the principles and Laws of Nature in the form of Uncertainty. This opposite forces or duality create the potential to run the system. We often see opposite school of thoughts people. slamming and criticising others to prove their own point. But the highest level of intellectual maturity is the acceptance of this duality and existence of opposites at the same time. Yes, this Opposite is the source of energy potential that induces the current flow otherwise everything will stop. This duality is inherently the part of the Nature scientifically. The Opposite Schools of Thoughts who end up fighting and proving their side to be true and others false must acknowledge this inherent duality in Nature, Life & Markets!! It’s Many World as in Physics ! Relative & Complementary to each other. The Beauty of Nature/Life/Markets/Universe is Duality and that’s essential to create Potential energy to function . This has to be understood well conceptually and not easy to digest psychologically for all. 6.Human- Nature Paradox. The Paradox is Humans consider themselves different from Nature but they are themselves Nature. Just like Wave-Particle Duality exists in the Universe/Nature where Light behaves both as a Wave and a Particle. Similarly, there exists the Duality of Determinism(Predictability)- Randomness(Uncertainty). The World is Predictable and Random both ! Yes it seems paradoxical but that’s how the Nature functions . It's not paradoxical when we include relativity and time dimension to the system. Something’s that has been often historically missing We have to imagine that Nature is a Work-in-Progress. Things are NOT already statically SET and might not be PREDETERMINED so that one is trying to predict completely. Predicting the Markets or Universe is Predicting Ourselves as we are the Observers and part of the system ! Self- Referential Issue! It’s like Predicting a Car’s trajectory that would depend upon the forces applied then. One might not determine now what amount of forces would be applied then. Prediction of Markets could be for the non-influential players to some extent, but for influential large volume players, markets are created by their own forces like Large Volume Short Sellers who create their markets to fall through the spread of negative news and crowd behavior by their own action of shorting in large volume. They don’t PREDICT, They CREATE THE MARKET by applying and inducing forces themselves !**The reason forecast doesn’t seem to work often in markets and Nature much is that observers' forecast could also start affecting their actions leading to change in the forecasted results. It’s because Observers are the part of the System itself. It's like Self-referential thing in a sense … our forecasts affects our actions and hence tend to change that. Let’s understand this technically conceptually: 7.Proof why Forecasting Often Fails in Markets ! Paradox of Prediction/Forecasting: Let at time T= t , an observer(trader) forecasts about T = t+1 based on the information available upto time t. The underlying assumption is that things would be the as expected between t & t+1. But this forecast at time T=t starts influencing the actions of the forecaster itself as well as the others who are the Causal subset of the market itself from T= t to t+1. It’s like Cyclical Process. Present Action affects Future Forecasts and Future Forecasts influences the Present Actions !! So, the Actual Result at Time T= t+1 would be affected by the Actions done between t & t+1 upon the Earlier Forecast at Time T=t even if correct let’s assume for a while) . So,! This way the Actual Result at Time t+1 could be influenced to be different inadvertently from the Forecasted Result at T=t. Hence the Forecasting often fails in general . To Note that Forecasters are the Causal Internal Subsets of the System called Markets/Nature. Had they been independent and external to the system, forecasting might work. It’s Quite Scientific Self-referential issue as in Godel’s Incompleteness Theorems . If we look at the Quantum Mechanical Theories like Schrodinger’s Cat and all, Actually ,an Observer could influence the Result by Observation at times upto various degrees relatively. Predicting a system whose part is Predictor himself is like Predicting the Self It’s like Self Referential Contradiction. This has also been the foundational issue in Science/Physics of ever there would exist TOE (Theory of Everything) that would make the entire Universe Completely Predictable using the Deterministic Set of Equations ! Completeness means Determinism in the Physical Universe. Markets being the Subset of Universe/Nature essentially affected by the issue of Self-reference. If one doesn’t Forecast and hence doesn’t influence the action ,then it has more probability of being correct. So, if you want to be correct, don’t forecast ! If you want to be incorrect ,then forecast in advance. This is general context! The fundamental subconscious assumption behind the whole philosophy of forecasting and predictions rest on the assumption of Static world in which we live independently existing of Predictor. It's not Universal Truth like The Sun would rise in the East that they might remain independent of the observer. Even there is uncertainty regarding the exact time when the Sun would rise, angle ,or It would be cloudy tomorrow and we might not see even the 🌅 🌄 Sunrise ! We have to consciously understand that The World is dynamic Work in Progress! It’s not already Set as we inherently presume to be . We are trying to predict the Work in Progress where are are ourselves the mechanics involved in its construction. It’s analogously something like lifting the bucket by sitting inside it . Hence quoting a bit by going out of the line but that I think is quite relevant : , it’s said that The Best way to Predict the Life is to Create it . This statement has hidden Truth that We should focus on Creating the Present Markets for ourselves rather than Predicting in Future ! Like Influential Short Sellers they try to create the markets for them by influencing the herd behavior. They don’t Predict ! Rather they create ! Hence the Underlying Philosophy of Forecasting/Prediction needs to be revisited. The above explanation is related to the Technical Impossibility of Complete Forecasting/Prediction. This is also one of the prime reasons that why AI/ML based Static Models rests on the Philosophical Assumption which doesn’t hold in Real World Finance & their accuracy is NOT very high. There is Philosophical mismatch in the two . Note : But that doesn’t mean we should not try to predict or forecast ! That's essential but we must acknowledge the fundamental limits to the complete Predictability!! Definitely we can and should try to predict to some extent causally and it could be beneficial at times ! Proof By Contradictions of Predictability/Randomness of Nature & Markets . Let’s take an example .To prove this, let’s assume the two extreme cases : Let’s imagine by the logic : The World is Completely Deterministic and we know this. If there exists formula which makes the World Completely Deterministic….will the World not reach Singularity and stop functioning over the time ? Uncertainty creates opposite market forces as in Nature. Or the causal formula would change over the time in terms of parameters’ values which would eventually contradict the assumption that there is absolute static complete causality. The World is Completely Random : Let’s imagine the world hence Market is completely random, then everyone would start making effort as anything could happen randomly…this chaotic scenarios would make the world and market stop over the time. The Point is - there must be Natural Duality of Randomness and Predictability in Market& Nature to let it function or else the stage of singularity would take place! 8)Connection between Science, Human Behavior & Financial Markets. Markets are driven by Human Behavior as well which essentially follow the scientific Laws of Nature. Human Brain is run by Neurons, Electric Signals etc. Human mind is basically an Quantum equipment Market is essentially like a Quantum wave as the resultant of Quantum Human Brain waves of set of Traders/Investors in the Market. Infact Human itself behaves like a Quantum wave scientifically. At deeper level, one can scientifically realize that even Human behaves like a Quantum wave ! Physics has some fundamental difference with Human. Unlike Physical Non-Living Objects, Human Behavior has some different Behavior but they are indeed linked to each other. But basically Human Quantum behavior also follows the Laws of Energy and Nature as non-living Physical objects do in Nature. Technically, Human Quantum Behavioral is more advanced has variable Components unlike the Constant one for Non-Living Physical Objects, but they follow the same laws of Nature & Energy fundamentally. As in Physics, Non-Living bodies follow Newton’s Laws of motion of forces, Quantum Laws, Principle of Least Action , Feynman Path Integral approach. Similarly, Living Human object also follows these laws but in variable way unlike constant for the former. Human Brain is also run by Neurons, Electric Signals, Energy etc. Let me state that like Physics, Human Quantum world also has Inertia,mass,weights, momentum, force etc. Broadly speaking just like Classical world, Quantum world also has the equivalent concepts. Indeed, the Newton’s Laws of Motions, Laws of Energy,Principle of Least Action (But Least has different meaning in Human Quantum world) are all valid in Human world. And as a result, even Markets has Momentum, Mass, Energy, Gravity, Acceleration etc.What we call Stocks also have mass, acceleration etc. . They also follow the Laws of Science/Energy etc. Essentially one must understand that Market is a Quantum Energy system. The Trajectory in Quantum world is fundamentally the Path of Least Action like in Paul Dirac’s Physical theories. It has been traditionally told that Humans are different from normal physics. Indeed that’s true but even Humans also follow the Laws of Nature and hence the Markets. So, I must state that Quantum Human Behavior also follow Principle of Least Action, Laws of Forces & Energy etc. as in Physics but yes at deeper level than Non-Living Physics which remains usually constant !!] But I would like to categorically add that Path of Least Action in Quantum Human Behavioural World driving the market may not be the Sum over all the Path Integral as in Feynman’s Path rather here the Effective Path is the Path of Most Stability having the Maximum Likelihood ! And , for that reason markets have to be studied from that scientific perspective of energy, mass, momentum, acceleration etc. Infact I would explain later the scientific background of concepts and how far economic rationales are correct scientifically especially for factor investing and all. For example, I’ll explain what is meant scientifically for economic terms like Large, Mid & Small Caps in terms of Quantum weights. How that affect their causal dynamics in context of factors. Scientific factors Not Associational. 9)Century old Problem: The Deeper Connection between Financial Brownian Motion( Louis Bachalier) & Physical Brownian Motion( Albert Einstein): Boltzamann equations and Langevin Equation : Newton’s Laws Technically Markets are run by Human Particle/Waves Behavioral Forces. Human Behavioral Forces run by Neuron and particles/waves in the Human Brains. Since, these brain particles/waves are themselves quantum type particles, they also follow the Laws of Nature like Newton’s Laws,Quantum Laws etc. Human Particle/waves Motions driven by those quantum microscopic brain particles like neurons are the same as the microscopic particles in physical world. The cells of the brain include neurons and supportive glial cells. There are more than 86 billion neurons in the brain, and a more or less equal number of other cells. Brain activity is made possible by the interconnections of neurons and their release of neurotransmitters in response to nerve impulses. Neurons connect to form neural pathways, neural circuits, and elaborate network systems. The whole circuitry is driven by the process of neurotransmission. Human Brain System is basically an Organic Electric System run by Neural signal network. Even Consciousness is also state of matter as postulated by few Scientists! Both are natural particles essentially following the Laws of Nature. Hence there is no point of difference. The Human Herd Behavior is scientifically the same as molecular dynamics in physics. Hence Financial Brownian motion and Physical Brownian motion are fundamentally the same scientifically but not exactly the same. As I explained earlier that Human Quantum Behavior is also run by the same Causal Laws in Variable way compared to Constant one as in Physical Non-Living Objects. But Fundamentally Human Quantum Behavior is also caused by the Quantum Forces and they also follow the Path of Least Action but Least here is variable . Recently some Scientists in Japan have shown some similarities between Financial & Physical Brownian motion by studying the behavior of HFT Trade Orders. It’s no surprise because I’ve already explained and make the statement that Human Behavior also follow the Laws of Nature like Physical Bodies as both originate from the Particles/Waves only. Human Particles in Brain are also like Physical Bodies particles following the Laws of Nature but yes one is Non-Living and another is Living particles but they both follow the Laws of Nature in their respective ways. Hence, it is proven that Human Traders Behaviors(Financial Brownian Motion) in Market would /is essentially similar to Physical Brownian Motion ! There is no surprise . As more and more studies will be done experimentally by studying the Human traders behaviors, they would come to this conclusion over the time that they also follow the similar Laws of Nature ! Just like Classical motion of Newton’s laws , Quantum motion is also governed by Newton’s laws… Newton’s Law is Universal originating from the Laws of Energy which is valid everywhere. Infact Newton’s Laws are derived from Laws of Energy, Principle of Least actions etc..which is universal every in the Universe “Like Classical world Physical laws(which essentially originate in Quantum world, just like mass energy, weight, forces, inertia, momentum, entropy etc. all exist in Quantum world as well leading to Quantum motion of which Human Behavioral forces are a type… But test in Quantum Human Behavioral living world, forces behave more like variables unlike constants in the physical non-living objects. Hence, the Humans are essentially an energy system and this is the scientific reason, they have similarities with non-living mechanisms. I hereby humbly make the statement that just like Physical Brownian motion, Financial Brownian motion would share lot of similarities with slight variabilities. It’s indeed science and one can confirm these through the experiments in real markets. Recently it has been experimentally shown at preliminary stage but as these things are explored more in detail, they would find experimental proof of my statements that Financial Brownian motion originating in Human particles Behavior is the scientifically similar as Physical Brownian motion. In a nutshell, Living Objects laws are similar as Non-Living ones but with extra variability rather than constant making the former more complex in this context! Hence, the market should be essentially studied as an energy system.” I’ve already found so far in my own real-world experimental studies ! 10)Science of Energy behind Black Swan & How to deal with it in Real World ? Complexity out of Energy Mechanism As we have seen earlier, the Scientific Universe /Nature has Randomness & Predictability as inherent duality like Wave-Particle Duality. Despite the fact everything is governed by the Causal forces, there exist many Causal forces governing the dynamics and beyond a observer level of information relatively.In any system there will always exist the zone of Unpredictability as well as Predictability and they exist together. In any day to day system, there will always lie some information beyond the system. This is fundamentally linked to Godel’s Historical results conceptually. This external information beyond the system is the actual cause of Unpredictability zone. There will practically always be some information not available to an observer at any time which lie external to the system ! Black Swan emerges out of this zone of Uncertainty & Unpredictability which is inherent scientifically in Universe, Nature, Life and hence in Financial Markets. This would exist in relative sense not Absolute sense depending on the Level of information available at a time. Hence, Technically Black Swan is nothing but exists in the domain of “Unpredictability, Uncertainty”. Hence , technically the statement “ Predicting the Black Swan is Contradictory as it means Predicting the Unpredictable”. “Predicting the Uncertainty”. I again emphasize it depends on the Level of information available to an observer. So, what needs to be done in Real world to Black Swan type events. First of all, one has to understand that it scientifically originates in the Unpredictability/Uncertainty zone’s, Predicting the Black Swan means trying to Predict Unpredictability! Contradictory! If it were Predictable, it won’t have been termed as Black Swan technically. So, one has to understand the science of Uncertainty/Randomness . As explained in the Randomness section, even Randomness has deterministic hidden pattern and things do converge to their stable equilibrium state over the time . This is according to the Laws of Nature/Energy. I must mention that Black Swan is technically nothing but Large Sudden Energy Shock/Force where Humans Observers panic and lead to jump or crashes. It is both due to internal and external factors and both are inter-related. So, fundamentally it’s Energy dynamics like in Physics!! Hence, one has to technically understand the Laws of Energy in Nature to handle this sudden jerk !! It’s like Stability & Instability in the system. This is proprietary here. This is basically the dynamics of traditionally called Complex system / Complexity which is also Fundamentally the dynamics of Quantum Energy. So, this Energy dynamics leads to Complexity in the system and fat tailed and eventually the Black Swan So, Rather than wasting time to predict Black Swan events, one need to focus on managing risk to be able to survive/ benefit whenever they come. This can be done by attaining Convexity ! This is because post Black Swan ,it would converge to the equilibrium state. Those who blow up during Black Swan events are not risk prudent for example they are exposed to risk due to leverage or similar fragility etc. The key is to survive during Black Swan. Hence one needs to be Convex enough to survive . This is the part of strategy how we could be more and more Convex. 11).Background of Factor Investing Factor Investing has traditionally been nothing but the application of Linear Regression Analysis tools on the Historical Return Data. It’s typically associational rather Causal where Linear Regression is conducted to calculate some superficial not-necessarily reliable relationships like beta and so-called Alpha. Especially Error term is often ignored. Factors such as Value, Momentum, Size etc. are quite prominent based on some economic opinions which are not necessarily scientific. One of the most buzzing factors has been Value, which has not performed over the years recently causing a global fuss if really Value matters or it has died.. So, how Value is calculated ? So, traditional factor Investing believes in some Economic Rationale or not even that, relying on LS Linear Regression Analysis Statistical tools to deduce, Alpha Beta etc. But there are lot of crucial issues. The most prominent is that while performing the regression analysis, it assumes that different factors are like independent variables without mutual relationship. So, in other terms, Value, Momentum etc. are assumed as Independent Factors as the Independent dimensions without trying to verify if they are really independent or not ! This is itself a blunder !! Different so-called factors like Value, Momentum, Size etc are not necessarily Independent but highly causally dependent on each other. We will see this later in scientific terminologies rather than superficial economic rationale. Now, one can imagine that based on the backward looking Statistical tools like LS Regression analysis(which is technically like inbuilt sample testing backward direction of time !!), one claims to calculate flawed concept of so called Alpha, Beta etc and that too often ignoring the most vital error components which is the real source of information especially about black swan.. So, what I mean to say that mathematical and statistical tools are not always wrong but their blind application to real world scenarios is indeed blunder ,. One can imagine how serious this risk is where Billions & Trillions of Dollars are invested based on such misconceptions.. We will look into more details scientifically before that let’s first look at the foundation infrastructure of Linear Regression and Conventional Statistical Tools. 12).Application of Statistical Tools in Finance : Misunderstandings : Role of Time in Mathematics/Statistics in Real World. Statistics Needs to be Scientific: They are always Backward in Time: Causal Science can make them look forward in Time. Fundamental issues in Conventional Mathematical Tools in Finance : Classical to Quantum Mathematics Statistics is abundantly applied in the world of finance. Almost all the financial models so far uses statistics whether for risk management or prediction etc. But the fundamental issue with almost all the conventional statistical models is they all are backward looking in the direction of time. This ignorance of understanding the role of Time dimension has possibly made the Financial models like a scam in itself may be. Mathematics has always been studied independently of Time assuming absolute for all the observers. This Platonist view is also one of the causes of misunderstanding and misapplication of Statistical tools in Finance which suffers from Hindsight Bias. THE FUNDAMENTAL PROBLEM WITH STATISTICS IS THAT IT’S ALWAYS BACKWARD IN DIRECTION OF TIME !! STATISTICS NEEDS TO BE REPLACED BY CAUSAL SCIENTIFIC MECHANISM IN FORWARD DIRECTION OF TIME DRIVEN BY FORCES ! The Problem with History is that observer sees only one history out of many possible unobserved histories that could have occurred like in many world theories. This biasness risk could be dangerous in real world for back ward looking financial models and tools. Financial Statistical Models are always built in Past data in the direction of Time. At Time T=0, a modeller does curve fitting for Data over T< 0. But the Modeller doesn’t understand ignorantly or by inertia that T> 0 & T<0 are not the same in the world of Statistics/Finance. By looking Backward in the direction of Time, Modeller rules out the Randomness component which is present while looking in future way back then. Future Direction of Time has many possible paths randomly but Backward Direction of time has just One Deterministic path which actually occurred and on which Data fitting is done. This foundational blunder of ignoring the Randomness in Backward Direction of Time is the core of all the issues in Financial Modeling based on Statistical Tools. The entire estimation of Risk, Prediction etc based on such Statistical Tools ignore the vital role of Randomness which a trader actually undergoes while taking decisions in real world. Unfortunately, Out of many possible unobserved paths in future, the statistical modellers only takes the observed path in the backward direction of time. The fundamental issue with Statistics is that it is always done in the backward direction of time based on historical data. That’s where the role of Causality & Science comes in. Causality tries to look forward using scientific mechanism . Let’s imagine in physics, we are using statistical analysis of past trajectory to trace out the future Trajectory ? Does it sound awkward ? So, then how we do that in Finance ? Finance has to be studied like Scientific Physics and that Principle in forward direction of time . This ingrained psychology of statistical analysis in finance should be replaced by scientific Causal analysis of future where we could study Randomness, Determinism, Human Behaviour etc.. And yes, some investment and risk strategies are causal based on scientific principles. We will talk about it later. Hence Psychologically engrained Backward Time Looking Statistical Analysis must be discarded and Forward Looking Scientific Causal Analysis like we do in Physics etc. must be adopted. This would also be the key to Scientific Risk Taking & Management and dealing Scientifically with Black Swan type events in Real World. The fundamental issue with the financial models is that these statistics tools, methods are all backward looking in time. They are not forward. It’s a paradox/Contradiction in itself that Backward Looking analysis tools are applied for Future analysis in the dimension of time. The tools like Back testing etc are the subset of that unscientific understanding of Role of Arrow of Time in Statistics in Real World. That’s the reason why Finance has to be developed in forward direction of time as we do in Physics. Do we Backtest in Physics to predict the Trajectory of a vehicle /Car or we study the equations of of causa forces. This fundamental psychological change has to be brought in the world of finance where the trajectory of a stock price is studied scientifically by analysing Causal forces, rather Backtesting. Even for those who simulate Randomly in future like MC Simulations etc must understand that There could always be more future possible Scenarios in Real World than one can simulate using Computers or otherwise. This could be fundamentally related to Godel’s results. The point is Finance has to be studied in scientific way in forward direction of time like physics/science. That’s the way, one can have better understanding of risk-return in real world finance. For that ,one will have to understand the Causal concept of Uncertainty/ Randomness in Life/Nature/Markets. This is because by understanding this concept, one can take Scientific decisions to build the portfolio or otherwise.. Otherwise all those Statistical based tools like Sharpe Ratio, Drawdown, Correlation etc. out of historical data or observed data is just the tiny subset of all the possibilities and hence hugely misleading for understanding risk in future real world. Like in Feynman Path Integral approach of Quantum Mechanics /Paul Dirac Principle of Least Action, Explore all the paths not just the observed ones and each path is based on certain causality like action of that path ! I’ll explain later in detail. Now let’s look at the Fundamental Compatibility of Mathematical/Statistical Tools in Finance. Traditionally the tools that are often applied are Probability Theory, Statistical Expectations Operators, Euclidean Calculus(Stochastic) , Linear Algebra etc. Let’s look at their origins. These tools were originated by contemporary mathematicians when there was classical world developments in physics by Newton, Leibniz , Gauss etc.. over the time Physics travelled from Classical Newtonian world to Einstein’s Relativity theories to Quantum Mechanics etc..but the mathematical tools that originated to support physics or otherwise remained trapped in the Platonist philosophy of view which would remain constant and independent in Classical forms. But as long as they are applied to Classical worlds, they are fine but when they are forcibly applied to Quantum world say Quantum Human Behavioural aspects like Markets, Finance, Life etc. These fundamental issues start raising up that how compatible those classical world deterministic Euclidean space mathematical/statistical tools are with the Quantum worlds particularly Human behavior which drives the markets? Let’s take an example : First with Statistical Operations like Expectation E[] which are used to calculate moments, and the Probability Theorems. Are in Real World Quantum Human Behavioural Space-Time Expectation Operator Formula is the same as for the Motion of Non-Living Natural Celestial Objects ? To remind that LS based Regression was first formulated to apply for celestial mechanics. In Real Life, does this traditional deterministic probability theory or Statistical Operators hold true always ? No ! If we look into the Logic behind this, it comes from the understanding of Logics & Physics of those Legendary mathematicians. That time Quantum Mechanics, Relativity, Principle of Least Action etc were not known. So, to make the mathematical/ statistical tools compatible enough for the Quantum Human Behavior these Probability theory, Euclidean Calculus etc. Not likely true. There and to be developed Quantum Mathematics, Quantum Operators, Quantum Expectations etc. as they work in Human Quantum world. Say for example to explain the difference, in classical world, if observer conducts an experiment 10 times, it would get the same outcome, But in Quantum world/Quantum Human Behavior, one can’t expect the same Deterministic Outcome of Expected Results in 10 Quantum Trials. One can relate it to Markets which is also a Quantum System fundamentally. There would be Uncertainty in the Expectation / Expected Result. So, the foundational point is Quantum Human Mathematical Tools have to be developed like Riemannian Geometry was developed for Einstein’s General Theory Of Relativity. We see mostly financial people using the classical conventional deterministic tools to prove and disprove n number of results in finance and real world without understanding the fundamental discrepancy and incompatibility somehow. Then they complain about Non-Stationarity without understanding the hidden concepts in depth. Blindly applying those statistical/mathematical operators, tools etc. and blindly applying to make investment decisions. So, the fundamental requirement is to develop new mathematical & statistical tools operators for quantum human behavior and hence financial markets . This is because Classical Euclidean type mathematical/statistical tools might not be fundamentally compatible with quantum human behavior. Hence, it doesn’t make true sense to model financial markets, risk etc. In real world. This is the prime reason I’ve not used Mathematical/Statistical tools here in the paper much as I don’t find them much fundamentally compatible (beyond a point) in real world markets driven by quantum human behavior. One can’t explain the fundamental issues with these Conventional Conventional/Statistical tools using these tools themselves. The problem of Self-reference. 13).Regression Analysis, & Paradoxes Y = \alpha+\ \beta\ast X\ +\ \epsilon \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ E\left[\epsilon/X\right]=0 E\left[\beta/X\right]=\ \beta do\left[X=x\right]Let’s assume the Linear Regression Equation Y=\ \alpha+\ \beta\ast X\ +\ \epsilon X=\alpha_1+\beta_1\ast Y+\epsilon_1 Y\ ≔ Xβ+ Zλ+ u where λ≠ 0 and u is the white noise \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ X\ causes\ Y\ iff P\left[Y\middle| d o\left[X\right]\right]>P\left[Y\right] Where Y is the Dependent Variable , X is the Independent variable and α is the Error Term of the Linear Regression. X = \alpha_1 +\beta_1*Y + \epsilon_1 So, we can see that Association ignores the Fundamental Role of Direction of Time here. . It could be both way but Causation has direction like Cause and Effect at different Times subsequently.. ! To determine Causality, one will have to block all other paths/factors and test the direct effect ..for example…Stocks in different economic scenarios…or other company specific factors to test the causality … For example: The Sun rises in the morning and the bird sings in the morning .They are obviously correlated but to test if Bird singing causes the Sun to Rise one can test if the bird sings when the Sun doesn’t rise on cloudy days or if the Bird doesn’t sing, does the Sun rise or not… One would find that this is not the case. That means it’s association ! Like two cars running on road would be associational not causal and they could suddenly change the direction after sometime if roads diverge suddenly…So, misunderstanding those two cars relationship is causal could mislead suddenly! Similar is like Factor Investing. If certain factors work doesn’t mean they are causal. They could be misleading and could be by chance unless Causality is established ! If this is causality the absence of one would definitely affect the occurrence of the other event. If A causes B then if A doesn’t occur, B should be affected every time. That need to be tested. But one does have to accept that there could be many more unknown causal factors that one might not know which could be in the form of randomness errors…!! So, that’s why Randomized type algorithms (e.g. RCT, Algo wheel etc.) type approaches would be required at later stage to deal with them !! Y=\alpha+\beta_1X_1+\beta_2X_2+\beta_3X_3+\epsilon Point 1) In the first one variable Linear Regression Model (LRM), what is the most important term is the Error term(residual ). This represents the Randomness ,Uncertainty component . The error term is the TRUE Origin of the Black Swan events & hence Disastrous Tail Risk. What is to be noted that in the above Regression formula, X1,X2 etc. are independent variables usually ignored in Factor based Investment Models. For example, while doing Linear Regression analysis in Factors and hence to calculate So-called Alpha, why is it assumed that different factors like Value (HML), Momentum, Size (SMB) etc. are independent ? Are they really independent in Real World ? No , they are not necessarily Fundamentally Independent ! so how far this Regression Analysis application to calculate Alpha is accurate ? The entire calculation of Alpha itself violates the fundamental assumptions/requirements of Linear Regression . The assumption of Independence in Statistics is the fundamental blunder in Real World Finance affecting huge amount of money management in the finance industry. As I usually say Assumptions & Approximations could be more dangerous than Nuclear Bomb !! 1)Reverse regression Y on X and X on Y are different ! 2)Relativity of Beta when many independent variables X 1 ,X2 etc are included. Beta changes . Beta becomes different when new independent variables are included in the Linear Regression Analysis. Hence the value of Beta is NOT absolute rather relative ! Statistics is NOT wrong but Statistician needs to understand the statistical terms well in real world applications and their assumptions ! This error term which is often ignored is the most vital component in the real-world financial applications. This is where the Mystery lies !! The General Structure of Machine Learning Models Like the Regression and other ones is the following \ Y=f\left(X\right)+\alpha It’s all about finding the suitable functions f(\ ). Here also once can see that though f(\ ) could be many but the error term still exists which is the source of Black Swan Tail Risk. The key issue is while most approaches on finding f() based on historical data fitting, what should be the focus is on managing future error terms (randomness uncertain components). Little effort is made to focus on Randomness (Error), most effort is focused on Modeling Deterministic aspects and assuming Error/Random term would be expectedly zero or like that. This is what the core blunder is in real world applications. AI/ML models also need to change its focus on that . That’s where the generic fundamental issue of Non Stationarity in Financial Real World Data is ! No matter how sophisticated function based on these deterministic classical Euclidean type mathematical and statistical tools are developed, the problem of Non-Stationarity would remain because of Fundamental Quantum Randomenss. Hence, The Real Focus should be on the Error term of the Regression /ML Models which reflect that quantum reflection ! That’s true source of Black Swan, Tail Risk, Fat-Tailed etc.. That’s where the real world model development is needed…Managing the Unknowns and Random Components in the form of Error is equally rather more important from risk /black swan type events.. Market is essentially the resultant of all Human Behaviors ! Need not forget. In context of Black Swan ,it is said that Real World is more Random than usually understood Indeed that’s right ! But then Randomness is also not Truly Random..It follows Deterministic Laws of Nature. It’s basically paradoxical in the form of Duality in Nature, Life and Markets. One more fundamentally important point is that Stock Prices don’t move on Euclidean Space of Paper in reality . This is virtual mapping of trajectory because Conventional (Classical)Mathematics itself has assumption about its own space . In Real World, the Stock Price moves in different Non-Euclidean /Riemannian/Some other Quantum Economic Space-Time which captures Human Quantum Minds etc.. Then we are trying to superficially/virtually draw the trajectory on the two dimensional paper . So, while doing Linear Regression Analysis , it must be independent of the intrinsic characteristics of the underlying space. Hence, whether Regression Analysis is done on Euclidean or different Geometric Space (as in Machine Learning etc.) the true physical relationship must be invariant ! We also often see AI/ML tools relying on Euclidean metric spaces only! That could be misleading inreal world finance. Hence, these parameters like Beta,Alpha etc depending on the slope of line in the Euclidean Space could be fundamentally misleading !! Because in different Space, Beta (calculating using Euclidean Metric Least Square Distance )could be possibly different but it has to be independent of the underlying characteristics of Space. This is extremely fundamental issue while doing conventional Linear Regression analysis/AI/ML models etc. . In a nutshell, this Beta relationship couldn’t be in the Real Economic Space Time but the characteristics of the space of the paper on which we have superficially assumed to draw the trajectory of the stocks by mapping. This is also because these conventional mathematical tools deriving out of Euclidean type metric spaces are fundamentally Incompatible to the Realities in other Quantum Spaces including Huma Behavior and hence Markets. 14)REGRESSION PART2 A)Causation is in the dimension of time which LS Linear Regression doesn’t take into account….Time dimension…it treats them statically in timeless dimension only space ! We have to understand in real-world perspective that Causation is established in the Space-Time dimension not just Space. As in the Physical world, cause occurs at say for example T=0 and effect occurs at sometime in future T= t. If we ignore the dimension of Time and take only Spatial locations into consideration, it becomes just association not causation. For Causation we have to test the Cause and Effect in Space-Time not just Space. Unfortunately, LS Regression is being conducted particularly in Factor Based Investing and otherwise in Finance ignoring the Time Dimension. This omission of time makes the entire relationship like associational of patterns that could be just co-incidence but not scientifically causal. By running the following LS Regression Y_t=\alpha+\beta X_t+\epsilon We often inherently assume that X and Y both are able to transmit information causally “simultaneously” at time T=t at more than the speed of light, but how this is possible ? If they are really causal, then first X should occur at Time T=t and this should cause effect to Y at some time in future T = t+n where n >0. But in the conventional LS regression it is inherently assumed that X and Y both are cause and event at the same time which is contradiction in real world scientifically in Nature unless it’s the case of Quantum Entanglement like events !! It otherwise proves that X and Y relationship is Associational based on Superficial Patterns either caused by Coincidence(Like two independent Cars moving on the road in the same direction misleading an external observer to be causal to each other) or they are caused by some hidden causal mechanism of some other variable known as Confounder. The above LS based regression can take place at the same time “Simultaneously” iff there is hidden Confounder(if it’s not coincidence!) as information can’t travel at more than the Speed of the Light ! (Of course it’s not Locality-at-distance logically here!) So, for Causational Proof, the Equation has to be in the form of do calculus. This is causal intervention that if X is causally set to value x at time T=0 for example how the effect on Y behaves in future value of Time =t and that time t can have different values on case to case basis. do\left[X=x\right]\ represents\ intervention\ where\ X\ set\ to\ x \ I have been advocating for long that Time Dimension is often ignored in the Statistical and Mathematical world(Timeless Platonic world) unlike Physical world(Space-Time) at the foundational level. The role of time appears non-sensical often but at deeper level it makes huge difference! Many of the long Paradoxes in Mathematics originate also due to this omission of Time as well fundamentally. Say for example, Theory of Relativity considers “Simultaneity” as Relative while Conventional Set Theory in Mathematics rests on the Principle of Simultaneity as an Absolute Phenomenon. This is beyond the scope of this paper. But I mentioned to show how this foundational misunderstanding percolates down to statistics and Association in LS based Regression is often misunderstood as Causation ! B) Let’s go back to the history if LS Regression method. It was first formulated by Mathematicians like Gauss, Legendre in 1722 to around 1800 for the estimation of Trajectories of the Celestial Physical Bodies like our Earth in the Euclidean Space. That’s mathematically good estimation as that physical space-time behaves like Euclidean Space-Time. But the foundational issue arises when that Tool from Physical Space Bodies is being applied to the Financial & Economic Space-Time. The financial and economic space-time driven by quantum human minds is not essentially an Euclidean Space-Time rather they have at least Non-Euclidean/Riemannian or some other Quantum Metric in Real World. Statisticians import the observations from that Space to Euclidean Space of Computer Screen or Paper to draw and apply LS Regression ! But this raises fundamental question that the Euclidean Space of Paper and Metric calculation doesn’t bias the Estimation of Financial & Economic Variables from a different Non-Euclidean space-time ? I am stating this because the way Least Square method is developed, is it Not inherently dependent on the Euclidean metric relationship ? What if these data are drawn on some other Non-Euclidean Space ? I mean this Relationship should be independent of the Underlying Geometry of Space where these are drawn. The Real -World Economic & Financial Space-Time is not Euclidean !! This a point of exploration as even in ML methods, often Euclidean metric tools are applied. But it could be Non-Euclidean and other Riemannian Metric Spaces as well to better discover the relationship. But in reality Financial time series data originate in Quantum Human Behavioral space-time. But anyway let’s confine to the Euclidean one that is generally taken to derive the Least Square based Regression Method for a while …. Least Square based Regression Tool is due to the algebraic structure of Least Square Formula where it minimizes the Sum of Squared Differences(we can say them Euclidean Errors!). If we change this objective of Minimizing the Squares to something different, the entire Calculation of Beta would be different and the value of Beta would change ! Beta is actually dependent on that Euclidean method. But why we minimize the Sum of Errors is itself under question ! How far is this process effective to take into account the Outlier in Real World ? Let’s say we don’t minimize the Sum rather we minimize some other functions that could be more suited to Outliers for example from Tail Risk Perspective. There could be different values of Beta on the same set of Data the way we define the minimization function of errors. It’s not Absolute ! It’s Relative ! The important concern is that in Factors Investing and Finance & Risk Management LS Beta is blindly applied for Allocation, Risk estimation that has proven to be disastrous in Real World in case of extraordinary(outlier) scenarios. Further, one more important aspect of LS based Regression is the Exogenous Condition that E[error] = 0 And if this exogenous condition of error is not satisfied, the whole estimation of Beta is unreliable and biased ! This is the fundamental reason why in Real World Finance, Beta is often biased and variable in Financial Time Series data. Exogenous condition means the independence of Error terms E[error/X] If Exogenous conditions are not satisfied, that means the Error itself is the function of some hidden relationship . It could be that Error is dependent on Y or X or some other hidden variables which govern the dynamics of the error and which in turn make Beta unbiased and unreliable as both are mathematically related ! Hence the internal dynamics of Error has to be established otherwise it would affect the other aspects of the LS Regression. Even there can be different dynamics of Error say if Error itself is some other variables regression Y=\alpha+\beta X+\epsilon\ \ where\ \ \ \epsilon=\alpha_0+\lambda Z\left(orXorY\right)+\ \epsilon_0 If such is the case \beta value would be highly misestimated ! This actually happens in the Real World Finance & Economics Time Series data ! It’s like in algebraic equation y=\ m\ x\ +c\ where\ constant\ c\ is\ itself\ a\ function\ of\ \ variables\ x\ or\ y\ or\ \ z\ldots In this case the m would be highly misleading ,biased and incorrect mathematically.! Infact such conceptual mistakes highly mislead in the real world finance and economics where beta(technically slope) becomes too variable dependent on the data time frame etc…and exposed to Black Swan Tail Risk affecting Billions/ Trillions of Dollars of Investment globally affecting common people and Investors’ lives. Error basically means Random Component ..all the deterministic components of Y dependent on Independent variables like X have to be removed from the Error term. So, In Financial /Economic world, unless Error satisfies the Exogenous condition E\left[\epsilon/X_t\right]=0, the LS estimate value of \beta will not be unbiased. If the expected value of the Error term above is not 0(the violation of Exogenous condition ), it means that there is still some hidden deterministic variables relationship to be discovered inside error terms. The error term like constant as in the equation y=\ mx+c\ is not a constant but a function of some hidden variables or even x\ or\ y\ itself. The correct meaning of \epsilon\ \ error\ term is that uncertain Portion of Y\ which\ is\ not\ explained\ deterministically\ by\ X. In Real World example the error term represents that Unpredictable component when we try to explain Y in terms of X causally. It's like we intervene by the cause do\left[X=x\right] at say Time T= 0 and measure its effect on Y\ at Time T= t. Infact in true causal sense these two occur at different points in time not at the same time(simultaneously) because information traveling takes some time. Then we try to understand what uncertain random component of Y is not explained by the deterministic components of the equation. We call this as the error \epsilon causally. But this omission of Time dimension makes the entire thing superficial associational relationship and just an artifact of Euclidean metric space !! The way error is defined as Y – E[Y/X], it shows that the error term is not independently defined. This is like B1 Spurious case as explained. “This is most likely valid in the field for which LS Regression was formulated in Celestial calculation but not often in the Financial and Economic Space. The Real World Data hardly show the Exogenous condition Satisfied.” Econometricians can’t define error deriving from algebraic equation like Y- beta*X rather error terms should independently satisfy exogenous conditions E[error ] = 0 Error should be independent of X Exogenous conditions means Regression assumes deterministic set up. Expectations of error or randomness to be 0 Correct meaning is if we do[X] at time t= 0 then at time t >0, Y should equal to X*beta. Error means Unexplained part in Causal relationship of Y due to X. Most Machine Learning Tools focuses on the deterministic part but there is need to focus on the Randomness(error) part and its underlying mechanism and how to manage those unexplained, uncertain components. Present regression method at same time means it assumes the role of hidden confounder. What is important to observe that in LS regression, the order of variable is also very important. The two regression lines below are not derived from each other with respective coefficients. Y_t=\beta_0+\beta_1X_t+\epsilon_t X_t=\lambda_0+\lambda_1Y_t+\mu_t That means in general LS regression method -\lambda_0\neq\beta_0/\beta_1 \mu_t\neq{-\ \epsilon}_t/\beta_1 \lambda_1\neq1/\beta_1 This reveals the internal dynamics of LS Regression that the Minimizing the Sum of Squared Errors method is not symmetric in this sense. The Order matters and at the same time the values of the Slope and the Error terms depend on the error and are not inter-linked. This implies that in real world financial and economic applications, before applying this associational LS regression, one has to make sure about the Order otherwise the derived slopes and randomness error would change leading to different estimations of risk affecting allocations etc. This also shows that Error itself is relative and order of variable dependent on LS based system . But in the Real World, the Random Components should be linked to each other if the Variables are the same just order changes. Coming back to the Commentary : the entire system is getting misled superficially by the Associational relationship based on LS type Regression affecting Trillions of Dollars Globally. The way sum of squared error minimization has been defined in LS system, it’s dependent on the Euclidean type system. Econometricians assume they are doing Causal but the mathematical tools used are associational. If LS system is changed, Beta will also be changed.. Question why LS system ? Rather than minimizing the sum of the squares, there can be other better approaches as well from tail risk perspectives. (This is the other detailed research topic in itself !). There is no fixed Beta . The value of Beta would depend upon how the Error terms are dealt with like LS equation. This is because Beta is derived by minimizing the Sum of Squared Error Terms. Lets say there are different points \left(x_1,y1\right),\left(x_2,y_2\right),\ldots.\left(x_n,y_n\right) We are trying to find the regression equation f\left(\alpha,\beta\right)=\sum_{n=1}^{n}\epsilon_i where \epsilon_i=y_i-\left(\alpha+\beta x_i\right) Here the values of \alpha,\beta are those which minimizes the function f\left(\alpha,\beta\right) It shows here that at time T=0 the error(randomness) terms are figured out, then at time T= t >0 they are minimized mathematically to derive beta (deterministic)term ! This is also like In-sample derivation of Beta term(Deterministic Term) from the mathematical minimization of function of Random Terms! Please imagine the role of Time dimension here. How Time dimension is often ignored. This is the fundamental cause of biasness. The LS Regression is basically like In-Sample Testing !! Lets say the Regression equation Here it shows that Randomness Term is the Function of Deterministic Term and that’s why we take the derivatives to find out the minimum. That mean at Time T =0, we try to estimate the Error first and then calculate Suitable Beta at Time T= t >0 , Please note the role of Time Direction as well. This is because for Causation, role of Time Dimension is the fundamental requirement in Real World. By the way, This is like In-Sample Estimation where Beta is searched by deriving from the Error terms. Ideally what happens in the Real World… We should pre-estimate Beta(Deterministic Expected Term) at T=0 and then measure Error Terms “INDEPENDENTLY” at T = t > 0 and study the Exogenous Condition E[ error/X] =0 if that is satisfied or not ! In the existing previous approach, “INDEPENDENCE” is completely compromised as Beta is derived from the Error itself !! This is in principle foundationally incorrect leading to Biased estimation of Beta in In-Sample way. So, the entire traditional Beta Estimation in LS approach is like In-Sample. It doesn’t tell about Out-Sample Beta in Real World in Forward Direction of Time! Especially for the Financial Data which is often unstable, this could lead to huge misestimation in real world out sample result. As we have seen earlier, the LS approach was formulated for Celestial Physical Bodies which is Classical World Stable Data unlike Financial Data driven by Quantum Human(Trader & Investor) Minds Exogenous Condition : LS Sum of the error minimum when Expected (error)= 0 . LS approach relies on the Expected (Average ) figure where Average is hugely misleading Statistical Tool in Financial world with fat-tailed data. The issue is even Exogenous Condition is not sufficient in Real World Finance because even if Expected i.e. Average is 0 satisfied but there is large negative movement and then subsequently large positive movement but the system could get exposed to Tail Risk Bankruptcy in the large Negative Error term and can’t even wait for the next Positive Upside Error Movement. So, from that Perspective of Tail Risk, even Exogenous Condition is not sufficient. It’s another detailed discussion of its own how these conditions should be re-framed ! We will talk later that Non-Stationarity is basically reflected in that Random error terms. 15). Beta is not Absolute but Relative. Beta depends upon how the error is dealt with and also with respect to what other variables are . The Exogenous condition of error is extremely important. So, probably one test is Take Beta to that level until Exogenous condition on Error Randomness is satisfied independently !!Then only Beta would be unbiased. Otherwise, it means that the Beta is biased as Error Term itself is some function rather independent ! As long as the error doesn’t satisfy exogenous conditions independently, it shows there are some more factors(contributing to different betas … deterministic components) to be discovered…. Moreover Beta value depends on how Error Term formula is tweaked like in LS form, There can be other forms as well. Beta would be different in various cases. It’s all dependent on the method followed. Beta should be such that it should maintain a balance between tail outliers and normal depending upon the requirements. If Error has intrinsic hidden further pattern ..it would make the existing beta biased and different types of structure within error would affect the dynamics of beta and further may be causality or not. 16. Hidden Dynamics of Error Terms : Now Further, There are different possible scenarios….Error terms have some hidden Deterministic components which affects X or Y or X affects the error term or even Y affects the error term …in all different cases the validity of Beta would be different Say for example Y_t=\alpha+\beta X_t+\epsilon But if Error term is biased and Not Random say for example \epsilon=\beta_1Z_t+\epsilon_1 And further Z_t\ \ is\ function\ of\ Y_t\ or{\ \ X}_t\ or\ some\ other\ hidden\ variables. In that case, the entire estimation of \beta would be highly biased as this is derived from \epsilon which itself is the function of different variables implicitly rather than independently. Biasness of Beta It’s like three variables algebraic equation in x and y and z..one is assuming z as constant and make quadratic in x and y…it could be technically wrong !! Similarly until error term is exogenous and independent…..it would lead to wrong beta estimation !! That’s why in Regression.. independence is of huge importance. If Error is function of some hidden variables then Beta won’t be correct…how do you make error is independent ? Simply doing regression without independence of error is technically wrong and could lead to biasness and misestimation. Like in Quadratic or Functional algebraic equations if you treat a variable as constant ..it would give wrong solution Y = m X + c ….here c needs to be strictly constant ….can’t be a variable related to X and Y or else m would be incorrect… One can’t treat function as a constant and do functional algebra ..it is technically wrong So when doing such statistical regressions, make sure error terms is not a function of some related variables or else beta would be biased and unstable over the time…and as beta would depend on error That’s the reason Beta becomes unstable in real world finance. It can be tested in real world where we see that the measured value of beta keeps on changing rather than fixed constant ! One can experimentally verify this on many financial time series data and check how beta has its own inherent dynamics. Beta keeps on changing dynamically over the time dimension or otherwise relatively if new variables are included! Error would change over the time if it’s function ..not exogenous conditions satisfied… If Error is itself a function not satisfying exogenous conditions ..it could be the source of Black swan and disastrous tail risk 17)Error Terms & Simpson’s Paradox : Simpson paradox reveals a lot about the Reality ! A) It reveals that the DETERMINISTIC parameters like Alpha, Beta themselves have Randomness over the time ! B) Further, the existence of these parameters and error terms are RELATIVE !! THE LINE OF BEST-FIT itself behaves geometrically RANDOM over the time ! What was the Direction of Error Terms eventually becomes the LINE OF Best Fit and what was Line of Best-Fit becomes the Error ! This reveals the Fundamental aspect of Relativity and Randomness over the time for an observer. It reveals the Fundamental Duality of Randomness & Determinism existing in the Universe,Life and hence Markets. The Universe and hence its subset ,the Markets have both the components: Fundamental & Deterministic. One can’t differentiate between the two. They exist Relatively and Simultaneously. Deterministic(Line of Best Fit) becomes Random(Error term) and Randomness(Error terms) becomes Deterministic ( Line of Best Fit) One has to be very careful while making decisions based on the Regression analysis particularly ignoring the Error terms ! Blindly following the Fixed Line of Best Fit to make investment decisions could be highly misleading and disastrous in real world !! So, this points towards extremely fundamental points while doing the Linear Regression analysis : That parameters like Beta, Alpha etc. could keep on changing depending on the data set and there are no fixed values. First, Two things tracing the same trajectory might not be correlated in true sense. Say for example two cars moving on the road would appear to be correlated to an external observer from sky. But that’s not so, may be later the road diverges after a long time and both the Cars could change the directions and befool the external observer from the sky who was assuming them to be correlated by observing their past trajectory for a long time !!. Similarly in markets !! The Point is to establish the Scientific Causation first and then one can rely scientifically Regression to some extent as long as error is managed well. For that causation, one has to block non-causal paths by observing the Y & X while blocking other variables or checking how Y behave if X is there and NOT there. If X causes Y then if X stops, Y must be affected or stopped ! We have to understand that Regression is fine it could help in prediction at times but given the large unexpected error terms at times could be highly misleading at times and disastrous. So, what one should do optimally is to expect along the Regression Line of Best Fit but still be prepared to manage Random Error term which could be huge in the world full of Complexity 18.LS Regression, Beta, Error, Simpson’s Paradox, P-Hacking, Reproducibility Crisis – Part 3 I think Black Swan Theory itself could be exposed to Black Swan without some Causality –Self -Referential Problem !! Causality is to be mixed up to diversify the Black Swan concepts In this Physical Universe where there is ongoing duality between Predictability and Randomness and consequently inherited to Markets. Human Traders/Investors are like Energy Particles driving the markets. There has to be a balance between Causality and Randomness approach. LS ..Square of Error is like Variance minimization…It could be Skew minimization or Kurtosis minimization or some other approaches ..for Tail risk. There can be different more robust approaches that can be explored in detail but beyond this paper as of now. Y_t=\alpha+\beta X_t+\ \epsilon X_t=\alpha_1+\beta_1Y_t+\epsilon_1 In LS method having two equations above …Randomness term would be quite different for the order of X and Y because the method to minimize Sum of Squares and Sum of Reciprocal of Squares would be different ! The reason why the coefficients in the above two equations are not interrelated is because Randomness(Error) term varies for the two equations affecting the Beta of the two equations. If the randomness (error) term is really constant like y\ =\ mx+c\ these two have been reversible into x=m_1y+c_1 where the coefficients could have been inter-related but that’s not the case. The fundamental reason is the way LS approach is done by Square of Error Minimizations and also the Error Term is not Constant rather some hidden function of variables, they are not inter-related in terms of their coefficients ! If really the LS regression is in the form y = mx + c where c is constant then the order of y and x would not change m and c i.e. beta and error terms. But in LS case error is not constant but itself a function. That makes it irreversible and beta and error both change on order reversal of Y on X and X on Y. It is because the LS equation is not Deterministic .It has randomness components as well and error determines beta by minimizing the square of error terms. So both are functionally related ..in LS frame work Randomness (error) derives Deterministic(beta) components! This indicates very fundamental issue while applying in the real world. One has to be careful why applying LS regression as to which if X has to be Regressed on Y or Y on X as the Beta and Error terms would be different and not inter-convertible respectively ! “ In the LS system , Deterministic Term(Beta) is derived from Random Term(Error Term) mathematically !” This also shows that Error is not independent of Beta and Beta is not independent of error. Deterministic term beta is the function of Randomness (error) term in LS which is found by the Sum of Square Minimization mathematically. This inherently means that Deterministic Term Beta is also derived from Randomness(Error term) ! And as in Real World especially Financial Data, the LS -Error terms keep changing being the hidden function of variables, the consequently Beta(Deterministic) term also keeps on changing affected by Error. I have explained this mathematically in the previous section. The foundation issue is Direction of Time …In LS regression analysis. Based on historical data backward time direction, we already know error(residuals as the difference calculated between y- beta x) ..making error as the dependent function on beta using the formula for residual..so error is no longer truly independent. But in Real World Forward time we don’t know the error first…we estimate y based on x and then uncertain component of y on x would be termed as error. so in physical causal terms ..we don’t know error in advance and then its expected value should be 0 ( exogenous condition) independent of X and y.. Need to minutely understand the role of arrow of time and independence of error from the deterministic term. It requires the higher level of imagination by learned expert traditional brains how the time dimension is ingrained in the mathematical and statistical developments that is often ignored and causes fundamental conflicts. So in real-world forward direction , these expectations (beta) and error(Random) occur at different order…first deterministic term expected and then independent error term… But here in backward time , LS …we know error (random component)already and then we derive beta(deterministic) part from that using sum of square minimization. True unbiased beta would be when independently calculated error term in forward direction of time satisfies the exogenous conditions. Not that we define the system backward in such a way using LS that exogenous conditions is forcefully satisfied and modify beta accordingly This is the crucial difference between causal structure and associational structure. Like in Physical World, Causation is in the dimension of Time and when we ignore Time dimension, it becomes Associational ! This is like we randomly assign do\left[\right]calculus do\left[X=x\right] as in RCT and then what Causal component of y is not explained by x ..that is error and should satisfy the exogenous condition in the forward direction of time This Randomness in forward direction of time is often wiped out while doing LS regression in backward direction of time So, in true sense Beta is not Slope in forward direction…Beta is made slope in backward analysis where error is already known…but in correct causal structure Beta is related to E\left[Y/do\left[X=x\right]\right]=\beta X In LS calculation Error(Random) term is made function of Beta(deterministic) term… Or Beta (deterministic) term is derived from the Error(random) term minimization… Infact this is generic issue with Statistical models, they are often in backward direction of time and static , there has to be dynamic and forward looking as the market is like an physical energy system, of course driven by human behavior which is also an energy system !! Statistics has to be made Causal forward looking by looking at it scientifically and dynamically. there the role of managing errors term is related to Convexity approach and Randomized Algorithms. Will talk about it later. Statistics has to be transformed into Dynamistics ! Will talk about that in detail in my book. 19. Paradox in LS- Regression Methodology : It assumes that Beta( Deterministic term) originates from Error (Randomness term) ! Hence Deterministic term is also Random if Error is Random…or else if it claims Beta to be really deterministic then Error term ( Random) is also Deterministic ! It’s paradox in itself ! This paradox is indicating towards the fundamental loopholes in the LS method of Regression how Beta is derived mathematically ignoring the time dimension in the biased way and if error terms are truly independent. In Real World Forward Direction, Causation world Randomness and Determinism are independently existing and not derived from each other Deterministically ! In Causal real world forward direction of time first expectations (Deterministic component is fixed ) then error(random component is calculated and then it must satisfy exogenous conditions independently if LS regression is correct and unbiased. In Associational backward time .. LS error term is figured out first and that using minimization of the sum function, the expected term beta (assumed earlier) is discovered by trial in backward direction of time and then changed to ! This traditional LS Regression is fundamentally like in-sample testing. Infact correct beta should be figured out by out-sample testing and checking which method should be taken compared to LS approach and how error terms satisfy the exogenous condition. Both the approaches are technically and fundamentally different. The Role of Time dimension is Extremely Important leading to this difference!! In backward LS regression, you derive beta(D term) based on Error(Random term)! ..In forward real Causal world, you fix Beta(expected deterministic term) and then derive the error (random) term independently.. I am repeating this time and again to bring to the conscious state of readers’ mind what they have been doing over last 50 years around. To use regressions for prediction or to infer causal relationships, respectively, a researcher must carefully justify why existing relationships have predictive power for a new context or why a relationship between two variables has a causal interpretation. The latter is especially important when researchers hope to estimate causal relationships using observational data The result of this In-Sample Testing is clearly visible when testing on the financial data how the measured values of Beta keeps on changing over the time unstably in real-world influenced by the error terms in the forward directions. Beta also behaves Randomly over the time ,which is supposed to be deterministic ! Beta change over the time and data shows that the Things are more Random actually not as much Deterministic as LS regression is expected to be. How can Physical things in 3 -4dimension be explained clearly in 2 dimension? How can Time dimension in Real Physical Causal world be explained in 2 dimensional Timeless Space ( Mathematical) ? Big Incompatibility…visible in LS Regression … Time (Causal) becomes Timeless(Associational) because of this reason ! That’s why Beta keeps on changing over the time, showing randomness not Deterministic aspect…Ref (Simpson’s paradox) This means LS Regression inherently assumes Randomness and Determinism are derived from each other algebraically using fixed deterministic rules…using LS method . ?? .but in Real Causal World is that the case..? No ! These things are extremely fundamental not be overlooked for this is the reason in real world linear regression misleads dangerously at times.. This misconception affects the investment allocations and all in the portfolio management and miscalculating the risk factor(beta) could lead to disaster ! In Real Causal World Randomness and Deterministic components come from different origin of information ! Key Point : That’s why if one uses LS Regression type in Real Causal World, one should simultaneously apply the RCT (Algo Wheel type ) method to handle Random (error term) along with Causal Deterministic terms. This model error and biasness in beta estimation using LS Regression has to be hedged by Strategizing the Randomness Error Term. Without Proper Strategy to Manage Randomness Error Term, it’s highly dangerous to trust Beta which is often ignored in day to day life financial investment decisions. That’s the important role of Causal adjusted Convexity to manage random error terms along with the Deterministic ones to manage disastrous tail risks and black swan events. Skill Vs Luck Or Managing Luck also a Skill ? It’s often said Skill Vs Luck to measure Alpha but I state that Managing Luck (by Strategizing Randomness) is itself an exceptional Skill !! Infact in real world finance Managing Error is the Exceptional Key Skill !! That’s where the role of Randomized tools tools like RCT,AlgoWheel etc. So, either assume Randomness and Deterministic are linked deterministically using mathematical formula or then LS regression is technically incorrect !! It violates and rules out the mechanism of Randomness and Deterministic only This means LS Regression shows the world has both Random or Deterministic components. But in Real World Random component has its own intrinsic determinism also but that’s not derived from the external deterministic components like in LS Regression. 20)Simpson’s paradox : Error and Beta in LS are Relative…Errors become Beta and Beta becomes Error direction wise in different reference frames. This also points that LS regression tries to determine the two dimensional system of points by one dimensional Beta measure while ignoring the direction of Error which is complementary to Beta. Or need to introduce two dimensional beta in place of one dimensional beta. Infact this shows that LS regression approach is not suitable for deriving absolute relationship. There is fundamental instability in Beta derived deterministically using mathematical formula from Errors. This also questions the validity of such regression in the real world. In another perspective, most fundamentally Simpson’s paradox shows the fundamental duality between Randomness & Deterministic terms interchangeably. To figure out the relationship between the variables. The measurement of beta in LS-Regression is Data dependent. & Time dependent and unstable especially in financial data. There is no fixed absolute relationship and the Beta (deterministic term) behave more randomly influenced by the error terms. In LS method…Beta derived from Error so in real world…need to manage error as well along with beta. It’s often taken beta and not the error terms, But the origin is error only in this flawed LS approach in real world Simpson’s Paradox reveals the importance for Causal relationship where one should reject non-causal betas and accept causal betas like extraneous values in algebra. It’s extremely careful thing to apply in social decision making as well where we get misled….by beta type relationships Rather than making Beta one dimensional…may be you can make it two dimensional. Along x and y coordinate axes…. *Two dimensional concept of Beta and Resultant rather than One dimensional* There would be dynamic equation for Beta and Error components as well Two dimensional pictures of LS Regression. Simpson’s paradox One direction analysis doesn’t fit for two dimensional movement of data ? Beta and Error are Relative..Time and Data frame dependent…not absolute So, it should be like Beta of the stock over last one month is 0.6….Beta of the stock over last 6 months is 1.2…like this… 21. Beta as the Time-Dependent Parameters in Real World Beta should be talked in the dimension of time like \mathbf{\beta}_{\mathbit{t},\mathbit{t}+\mathbit{n}} denoting the value for the time t,t+n for the financial time series data. Like Beta of Stock over 1 month, 6 months, 1 year ,10 years like that. This is because the values of Beta behave more randomly than expected influenced by the dynamics of the errors in LS regression especially for the financial time series data. Beta keeps on changing over the time unstably over the period of time. Hence simply representing Beta as the constant term independent of time dimension is highly misleading in real-world causal financial world ! This traditional definition and representation of say Beta of a fund is 1.2 for example is fundamentally misleading and incorrect in the real world the way LS Regression is derived. The Fundamental Role of Time Dimension is Extremely Important which is ignored Traditionally in the Investment Industry. Since, the Physical World is Causal in the dimension of Time, Statistical & Mathematical tools used in the finance have to be essentially incorporate the dimension of time. LS Regression parameters ignoring the time dimension is ruling out the Causational and making it Associational by ignoring the Time Dimension. And, it should be mandatorily be followed the way financial investment and risk industry has been traditionally. Beta not Static as usually treated…. Simpson’s paradox --- Ignoring Time dimension of data frame…even beta has time dependent function….not constant as in LS….not to be treated as constant static as usually done….dynamic So, as it’s two dimensional thing, one has to study both the dimensions…(error and beta ) ..error as well.. Need to study dynamics over the time…not just static The most fundamental reason is Ignoring Time dimension in mathematics and statistics in Real World Physical applications where Time is crucial for an event to occur. Psychology and Social application of Simpson’s paradox Error is also a direction…. Errors and Beta are like X and Y axis… Should be Independent Dynamics of Statistics over Time dimension can’t get captured by the Static Capture of Statistics* Beta & Error are functions of time ..ignoring that is leading to many paradoxes and conflicts Simpson Paradox also shows the fundamental issues with LS regression method Taking investment and risk decisions in real world based on static view could be highly misleading 22). Simpson’s Correlation Paradox(Misleading Correlation) in Real World Say for example : X & Y show positive correlation over 1 year statically ..but they are infact negative on daily basis dynamics Or they show negative correlation on 1 year but they are positive on daily basis dynamics Simpson’s Contradictory statement --- Role of Causal Time Dimension & Correlations can take many different values over different time and data Real-World Judgement can have huge variation No value of Beta without Error terms Simply taking out Beta(Deterministic component) (direction) hugely misleading* *Hence handling (Random Components)Error is crucial and complementary* It’s two dimensional thing so taking one and ignoring the other could be dangerous* P-Value Hacking : P-Value itself has statistical distribution as a stochastic/random variable. Most often the favorable P-value of the distribution is selected in a biased way. Which P-Value is being selected out of its own distribution is a big concern ! Exogenous condition: E(). = 0 but in real world if Errors are too large positive and Negative then also E( ) = 0 but that doesn’t mean it would work in the Real World as any Large Error can bankrupt the system even if average is 0* Least Square Method is basically Minimum Variance Principle but Variance may not be the true statistical representation always especially in real world finance. Hence just Exogenous Condition wouldn’t be helpful in the Real World!! Even if E( ) = 0 but it could bankrupt in the Real World. What matters is not the Expected Value rather the dynamics of error term. Later on we will also see that this definition of Expectation Operator could itself be fundamentally flawed in real world. Beta is meaningless unless Error is managed well !! Deriving Beta from Error (Residual) and then Calculating R^2 by involving them ..it's totally Insample thing ..even high R^2 can’t be relied as Beta is derived from Residual only… P-value Hacking : Backtesting Randomness Best Cases selection based on Random Parameters testing…. Misleading P-Value,R^2, Correlation as Random Variables in Real World. P-Value ,R^2, Correlations all are themselves like Stochastic/Random variables having their own statistical distributions, hence their values are not fixed rather variable. Hence their conventional applications in real world could be highly misleading treating them often as constants. One can select the favorable mean P-value out of its distribution to false claim one’s result statistically. But what if tail values of P-value distribution gets realized ? This is similar to Correlation which also has its own statistical distribution and that’s the reason Correlation could be highly misleading in Real World especially Finance. These days while performing backtesting or simulation using computer based algorithms, so many trials are conducted and the best is selected to false claim the great performance of the strategy. This is basically Reproducibility Crisis in Finance. But what is often forgotten that every Random Trail must come with some Cost in Real World to remove Selection Biasness. Will explain that in Convexity Section later that how Causality can minimize that Cost ! In context of Reproducibility Crisis That's not Independence .It should be like you take random trials and then see how do you perform...not that try 1000 trials and select the best....Best varies over that time....Best at Time T =0 might not be equal to Best at Time T = 1 Even the “Best” could be having Randomness and Time Dependent. It's like in sample result to select the best and leave the rest. Will explain later in this article on Convexity that how each Random Trial should have some cost involved and how it’s related to Reproducibility Crisis !! 23).Equity Risk Premium Puzzle : It’s indeed the bug in Factor Investing statistical structure based on the Linear Regression Analysis which we have discussed earlier…Capturing Human Behavior dynamics based on Linear Regression analysis deterministically while ignoring the randomness component is deeply flawed ! This puzzle arises because we misunderstand the concept and foundation of Regression Analysis.. Any regression based model to determine this risk-premia will have fundamental concern… This is simply related technically to the fundamental issues of randomness and structure with such Regressions on which CAPM and Factor based Models are based… the foundational issue is in the statistical tools itself on which such model puzzles have been developed !! So, the ultimate solution to one of the most important historical puzzles in finance is to look into the flawed conceptual misunderstanding of the statistical foundation which has given birth to the so-called puzzle. The origin of the Puzzle is the loopholes and misunderstanding in the statistical infrastructure which has given birth to such superficial puzzle. Once that is understood, the puzzle would cease to exist automatically ! So, this puzzle infact hints to revise the foundation ! I must again categorically mention : When we get the extraneous solution of any mathematical equations, it’s not that mathematics is incorrect rather the assumptions while applying the mathematics are incorrect. Similarly, here we need to revisit the foundation of the flawed assumptions of factor investing and the statistical infrastructure like Regression. Coming To Factor Investing Analysis : As the Byproduct of Regression Analysis Equity Risk Premium Puzzle : The main reason behind this Equity Premium puzzle is fundamental structure of the Regression based CAPM model, Misestimation of Risk Calculation and Ignoring the Time Dimension. Often Risk is measured in terms of Beta based on Historical data in backward direction of time using the same flawed estimation of LS regression techniques. Even Traditional Tools like Max Drawdown etc doesn’t capture the true Risk in the Real World. The fundamental reason is Time dimension is ignored. If we look in forward time direction, there could be many possible scenarios that an investor is willing to take while investing in Equity but not all such possibilities actually occur. Say for example an investor in the time of Uncertainty is willing to take risk of even market falling -50% say due to some war or some viral outbreak etc.. But somehow, market didn’t fall that much and only -25%. So, when regression is done on historical data or even max drawdown is calculated on historical data in backward direction of time, this possible forward scenario of -50%(which didn’t occur) was not included ,which an investor actually took in forward time uncertainty. Existing Conventional Tools based on Regression Analysis and Max Drawdown, Sharpe Ratio etc. often don’t take into account the possible future real risk which didn’t occur but was a possibility in forward direction of time ! infact that was the real-time risk an investor took in forward time direction , but was omitted in backward analysis of time ! This could hugely underestimate and overestimate the risks. So, the fundamental reason of Equity Risk Premium is the Misestimation of Risk in Real World by Ignoring Time Dimension and Traditional Backward Looking Tools e.g. Beta, Share Ratio, Max Drawdown etc. Possible Randomness in Future gets omission in Backward Analysis! Fundamental Issues with LS -Regression based approach (the backbone of Traditional Factor Investing) Also, the omission of Fat-tailed Outlier data which could drastically change the estimated risk parameters like Beta etc.. Hence, unless the Risk is Truly measured in Forward Direction of Time( Not on Historical Tools), the true estimation of Equity Risk Premium would be inaccurate. You are measuring the estimate based on back ward looking data….in forward real world risk is way more than backward looking due to Uncertainty… *Because equity risk premiums require the use of historical returns, they aren’t an exact science and, therefore, aren’t completely accurate.* The estimates also vary largely on the time frame and methods of calculation…removal of outliers fat tailed negative data Relying on CAPM type regression model to do the historical analysis for equity risk premium based on Beta etc.not reliable enough Risk should be truly seen in the forward direction of time not backward analysis of Historical data using regression models In real world we see how much risk we take for equity investing over the Risk-Free rate etc. Measurement of Risk in the form of Beta using Regression on historical data is not correct in RealWorld!! Real world Risk in forward direction of time is way more than the backward analysis of risk using regression tools Lot of randomness that appear in forward direction of time in real-world doesn’t get captured in the backward analysis For example : risky scenarios that could have taken place while taking investment decisions in fhe forward direction of time ..but say they didn’t actually happen …will not get captured and lead to less risk calculation on historical data regression analysis… .: Misestimation of actual risk in forward direction Risk is hugely misunderstood on backward Anlaysis and in forward directions..many disastrous possibilities had the chance which actually didn’t occur and hence don’t get captured truly in the risk measurement. An equity risk premium is based on the idea of the risk-reward tradeoff. It is a forward-looking figure and, as such, the premium is theoretical. But there’s no real way to tell just how much an investor will make since no one can actually say how well equities or the equity market will perform in the future. Instead, an equity risk premium is an estimation as a backward-looking metric. It observes the stock market and government bond performance over a defined period of time and uses that historical performance to the potential for future returns. The estimates vary wildly depending on the time frame and method of calculation. Future Beta & Historical beta An equity risk premium is an excess return earned by an investor when they invest in the stock market over a risk-free rate. This return compensates investors for taking on the higher risk of equity investing. Determining an equity risk premium is theoretical because there’s no way to tell how well equities or the equity market will perform in the future. Calculating an equity risk premium requires using historical rates of return. CAPM model itself needs fundamental change based on historical regression How exactly to calculate this premium is disputed How exactly to calculate this premium is disputed Equity Risk Premium = Ra – Rf = βa (Rm – Rf) But how do you calculate future beta in real world ? Not based on historical analysis By the way this is associational not causational … Expected equity risk premium likely depending on the hidden causal factors not just on this associational beta based on Regression where the role of confounders also come in. Future return Equity Risk Premium would actually depend on the Causal Mechanism & factors & Randomness not this associational type models like Factor based models and CAPM etc. Future Expected Fair Valuation drive the return causally scientifically Not this CAPM type associational superficial statistical theory One needs to further scientifically understand why an investor took systematic risk means he/she should be compensated with return why ? What is the science behind this ? This would be explored later. Missing Causal Confounder variable actually affects the equity risk premium. That is hidden but could make the entire risk calculation biased ! 24). Scientific Laws of Motion in Markets & Quantum Law for Markets : Newton’s Laws of Motion for Markets : Principle of Least Action by Paul Dirac : Feynman’s Path Intergal approach in Quantum & Classical World Laws of Motion of Stock Prices in Quantum World: Net Resultant Valuation(Energy) generates Forces(Demand * Supply i.e. Buy and Sell Orders i.e. OrderFlow which finally leads to the Motion i.e. Change in Momentum Newton’s Laws of Motion in Microscopic world : Consider a system of N classical particles. The particles a confined to a particular region of space by a "container'' of volume V. The particles have a finite kinetic energy and are therefore in constant motion, driven by the forces they exert on each other (and any external forces which may be present). At a given instant in time t, the Cartesian positions of the particles are r1(t),⋯,rN(t). The time evolution of the positions of the particles is then given by Newton's second law of motion: mir¨i=Fi(r1,⋯,rN) where F1,⋯,FN are the forces on each of the N particles due to all the other particles in the system. The notation r¨i= d^2 (r i/dt^2) i.e. second derivative. N Newton's equations of motion constitute a set of 3N3 coupled second order differential equations. In order to solve these, it is necessary to specify a set of appropriate initial conditions on the coordinates and their first time derivatives, {r1(0),⋯,rN(0),r˙1(0),⋯,r˙N(0)} Then, the solution of Newton's equations gives the complete set of coordinates and velocities for all time t. For an Investor , based on the valuation, the causal force of demand is created for a stock by the trader/investor. The Valuation is like Total Potential (in Physics) that generates the force in the stock. Valuation is relative depending upon the investors’ perspectives. Valuation (Potential) generates the market forces. Yes it’s as per the Physical Laws of the Nature. So, there would be different valuations for different investors/traders (who are like human brain quantum particles!) causing different forces. Let V_1t denotes that equivalent force caused by the respective potential (Valuation) V_1t,\ \ V_2t,\ \ \ldots\ldots V_nt V_{\left(resultant\ at\ time\ t\right)}=V_1t+V_2t+\ldots..V_nt So, the causal process : market is the system of different investors Each investor i has the relative valuation V_i. Every Non-Zero Valuation would apply either upward or downward forces. \mathbit{V}_{\left(\mathbit{resultant}\ \mathbit{at}\ \mathbit{t}\right)}=\mathbit{V}_{\left(\mathbit{upward}/\mathbit{buyer}\ \mathbit{at}\ \mathbit{t}\right)}+\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \mathbit{V}_{\left(\mathbit{downward}\ /\mathbit{seller}\ \mathbit{at}\ \mathbit{t}\right)} Causal Process due to Forces in the Markets. Different Valuation-generated forces V_i attracts and causes different investors/traders to put Buy/Sell Orders O_i. O_i≔Vi (i.e. demand and supply forces) .This is scientifically like the gravitational force on energy or law of attraction! Higher Value tends to attracts small values towards itself.. The Resultant Order Flow Imbalance is caused by the Sum of all these Buying and Selling Orders (Caused due to the Value generatedDemand-Supply Forces). {\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \mathbit{O}}_{\left(\mathbit{resultant}\right)} is like Electric Flux i.e. Number of Order Lines passing through due to Valuation Caused Field. (Note : I refer here only to the genuine orders intended to be executed not the false orders ,for that there will be more causal analysis) \mathbit{O}_{\left(\mathbit{resultant}\right)}=\sum_{\mathbf{1}}^{\mathbit{n}}\mathbit{O}_\mathbit{i} {\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \mathbit{O}}_{\left(\mathbit{resultant}\right)}≔Vresultant The Resultant Order Flow( due to Valuation Generated Forces) causes the Momentum change for the next time duration [Newton’s Laws of Motion] \mathbit{MO}\mathbit{M}_{\mathbit{t}_+}≔Oresultant \mathbit{Set}\ \mathbit{of}\ \mathbit{O}_\mathbit{i}\ \mathbit{determines}\ \mathbit{the}\ \ \ \mathbit{Liquidity} \ \mathbit{Liquidity}\propto\mathbit{Numberof}\ \mathbit{Opposite}\ \mathbit{Orders}\left(\mathbit{O}\mathbit{I}_\mathbit{i}\right)\ &\ \mathbit{Quantity}\ \left(\mathbit{Q}_\mathbit{i}\right) \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \mathbit{V}_{\left(\mathbit{resultant}\right)}\propto\mathbit{Liquidity} Now As the Stock Price gains Momentum due to forces, Price changes using F = m *a (Newton’s Laws of Motion type mechanism), The Price changes subsequently. Price at time t+ would be caused by Momentum generated at time t and then updated Valuation for different investors would change relatively due to change in Price and Momentum in the dimension of time. Price and Momentum being two of many factors affecting the Real- World Valuation at anytime. ( This has been explained in detail) \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \mathbit{Price}\left(\mathbit{P}_{\mathbit{t}_+}\right)≔MOMt \mathbit{V}_{\mathbit{i}_{\left(\mathbit{t}++\right)}}≔Pt+ & MOM_t Explanation : Valuation at any time V_i is the function of many multiples and factors and their dynamics over the time for every investor. There is a resultant of all the Valuation. Open-ended process. We have explained earlier \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ V_i\ \ \ \ \ \ \ at\ \ \ \ time\ \ \ \ t_+=f\left(MOM_t,Price_t\ldots\ldots.\right) So, to be specific related to Momentum and Value Relationship a) Valuation at time t is also dependent on Momentum at time t-1 i.e. previous momentum. The mechanism for Valuation is mentioned below.. V_t≔MOMt- a)Valuation at time t drives the momentum (velocity) at time t for t+1 through the force it generates. MOM_{t+1}≔Vt a)As the Stock price goes into the motion(momentum), it adds values for different investors leading to change in the Order Flow Imbalance. O_{\left(resultant\right)\ } is caused by more and more forces in the markets caused by the Valuation changing over the time. O\left(resultant\right)≔Vresultant at t OI_t\ is\ like\ Electric\ Flux\ i.e.Numberof\ Orders\ Flowing Number of Field Lines crossing is akin to Number of Trading Orders of Traders/Investors caused due to Valuation changes as the Potential. Which depends upon the Current Strength caused by the Potential/Force/Valuation changing over the time. Order Flow is essentially and scientifically the Market Quantum Force. One needs to study the Order Flow (Force) to be able to understand the motion of the resultant Stock Price. Like we study Newton’s Laws of Motion in Physics to figure out the trajectory of a body. One can apply ML to study Order Flow Dynamics to study the Quantum Forces driving the Market. It can be quite useful to study dynamics of tail events in fat-tailed events . This is Proprietary here further ! So, Based on the Laws of Quantum Motion affecting The Human Traders’ Behavior and hence the Stock Price can be studied in the ORDER FLOW dynamics how Forces evolve. This is where ML could be useful. I state and claim based on my experiments that it’s similar to Energy system of Particles. This is linked with Century Old Problem and essentially Financial Brownian Motion would be similar to Geometrical Brownian motion but in different sense ! This is linked to what I’ve explained in other parts that Human Brain follows the Laws of Quantum Motion and that’s the advanced version of Classical World Laws of Motion. So, microscopic Newtonian dynamics, the Boltzmann and Langevin equations are derived for the macroscopic dynamics. Hence I state and claim that one can experimentally verify that Financial Brownian Motion (Quantum Living version) is possibly more advanced form of Physical Brownian Motion (Quantum Non-Living Version) although their foundations being the Energy system only fundamentally but in different forms. This fundamentally originates from the fact that Human Brain Quantum Behavior is the advanced version of Classical Laws of Motion where the former is variable while the later is constant. But yes all these Classical Laws of Motions are valid and originate from the Quantum world only. That too, Living Beings Quantum Laws have more variable components than Non-Living Quantum Laws but all of them are highly Causal. This is beyond the scope of this paper. But for this paper, that can be well studied and verified in the behavior of Order Flow dynamics ! It’s fundamentally the energy dynamics. More the Valuations for different investors, it would attract more OI assuming there is availability of fund with the investors. To note : Momentum is just one of the factors among many. What matters is the resultant of all like in physical forces. These are similar physical forces attracting investors through their quantum force on their brains(neurons). Step2 :At any moment in time The V(upward/buy orders )tries to push the price up and V(downward/sell orders) tries to push the price downward. This is tussle between the two driving the market ,eventually converging to the equilibrium state as happens in physics. Step 3. The Momentum drives the prices further up due to the net upward force. Gradually based on the combined effects, the price moves. The distance travelled by the stock price can be calculated on the change in momentum caused by the forces. Step 4: As the price moves up and the momentum increases, it affects the Valuation dynamically. Valuation (upward ) and Valuation(downward) keeps on changing time to time. Upward force and downward force keeps on changing. .Gradually as the price goes up significantly , the Upward Force weakens gradually due to weaker upward Valuation and downward force becomes strong due to stronger downward valuation. Upward That leads to price coming down again. This cycle goes on to reach the state of equilibrium and fair value over the time. \ \mathbit{F}=\mathbit{m}\ast\mathbit{a} [Newton’s Laws of Motion] This is universal law in Nature not just for Physics of Non-Living Objects ! But yes in different forms because nature of motions differ in classical and Quantum worlds ! But this is also applicable in quantum brain world .. \mathbit{MO}\mathbit{M}_{\left(\mathbit{stock}\right)}\ \mathbit{i}.\mathbit{e}.\ \mathbit{Momentum}\left(\mathbit{stock}\right)=\mathbit{Mass}\left(\mathbit{stock}\right)\ast\mathbit{Velocity}\left(\mathbit{stock}\right) \mathbit{V}_{\left(\mathbit{resultantforce}\right)}=\mathbit{Mas}\mathbit{s}_{\left(\mathbit{Stock}\right)}\ast\mathbit{acceleratio}\mathbit{n}_{\left(\mathbit{stock}\right)} \mathbit{V}_{\left(\mathbit{resultant}\right)}=∆MOMstock Here Mass of Stock can be taken as Market Capitalization which typically means Small Cap Stock will have less mass than Mid Cap than Large Cap. So, why Small caps could be more volatile than large cap in general is also because of their lower mass and hence requirements of forces. But this would depend on the resultant forces on case to case basis. But yes it’s indeed caused by scientific equations of forces ! Like in Classical world, Quantum world also has gravity ! Larger stocks are having more mass and hence gravity than smaller caps in general comparatively. Yes it's like Physics . So, there is more amount of force required to move its price to similar distance (i.e. same return ) compared to small cap (low mass) stocks in general given all other things are constant. Hence they also show lower risk as more amount of force would be required to pull their prices down. And there this forces come from demand supply forces which are actually quantum forces in the brains of buyers and sellers ( traders) or computer codes… “Factors have to be discovered scientifically how they contribute to the Quantum Laws of Motion of Stock Prices not associational regression based approaches traditionally followed. One has to discover how a factor is affecting the acceleration/motion of the stock through Force and Mass ..For that one has to study the Law of Quantum Motion of a Stock Price originating from Human (Investors) Forces of Demand Supply & Order Flow. So, one has to study how Momentum , Size, Quality, etc. are contributing to the Scientific Laws of Equation . Not that they should be compensated for their risk due to a factor and hence the return! Hence, Linear Regression based CAPM result that Higher Risk means Higher Return etc may not be valid always in Causal Scientific Dynamics of Risk-Return. This Economic Rationale that an investor should be compensated just because he/she has taken some specific type of risks has to be scientifically validated and determined how they are contributing to the Laws of Motion of a Stock Price, otherwise it’s just a superficial and non-scientific and not true ! Infact explained later that Economic /Financial Space- Time itself exists scientifically having their Laws of Motion as we have in Physics!! 25)The Fundamental Issues with Conventional Value Method Connection with Regression Analysis Price to Book Value wrt. ROE (Ref. Aswath Damodaran papers). This Linear Regression Analysis is again applied with misleading results about Undervalued and Overvalued. Also, something becomes undervalued using one multiple while overvalued using another multiple. This also reveals the fundamental flaw and incompleteness of single multiple approach to Valuation. If one uses(despite flaws),one should possibly take into account different multiples to figure out the same. Infact there exist proprietary model based on AI/ML or based on the dynamics of various factors to decide the true Valuation in Relative Valuation approach which is so widely applied in Valuation for Investment industry. Traditionally we have seen factors e.g. Value, Size, Quality, Momentum etc. based investing being very popular and at the same it has also shown the dismal performance over the last few years. Two important aspects : a)Value as the term has been defined incorrectly in this context that Low price to Book Value mean Undervalued and High mean Overvalued or similar other analysis. Infact this decision on Low or High is based on the LS -based Regression Line as well . Given the fundamental issue with the Regression Line, the basic definition of Low and High could be mis-estimated. Moreover, even if one takes LS based Regression approach, Valuing something based on just one specific ratio could be misleading as different Ratios reflect different components of the Valuation(Relative Valuation Approach). Low Price to Book Value Vs ROE Regression(Reference Aswath Damodaran).Misuse of LSS based Regression to figure out Low to High Book Value. The determination of Low and High based on the Line of the Best-fit is itself faulty ! Again same typical Regression issue. Line of Best-fit itself could itself misleading in Real-World applications. So, what one figures out Low could be High if Line of Best Fit is changed ! There could be indeed better methodologies to incorporate different ratios dimensions of Valuation but that’s not the discussion of this paper here. Value in Real World is not the same as such Academic Theoretical Definition as conventionally calculated in the Finance & Investment Literatures. The Definition of Value needs to be Redefined and Recalculated even in the Literature. This is the fundamental reason why Value has started underperforming over the time recently. So, we shall discuss on this in detail below. This Theoretical Definition of Value is itself flawed and incomplete when we look at how Value works out in Real-World Markets for Investors. How Investors figure out value in a stock traditionally is itself flawed based on conventional approaches. There will always be some patterns in this world enough to mislead us and that could be due to chance. As assumed in Linear Regression based approach to judge the Value , in Reality, all these factors don’t independently exist rather have interdependence which creates Value in totality. We shall see this as we go further here. .The basic statistical method to apply and judge Alpha or decide the performance of Factor based approaches like Fema-Fench models is itself flawed leading to false calculation of Real World Alpha. 26). REDEFINING VALUE IN REAL SCIENTIFIC WORLD IN INVESTEMTN :REDEFINITION So, How this Valuation formulates in Real World. There is the difference between Academic Old Definition of Value and Real-World Scientific Value. Infact Academic definition of Value should be redefined. Here is the brief formulation: Now taking the idea of Relative Valuations Multiple here…Valuation of a stock for an observer(investor) could be the different (Relative Valuation). Every Relative Multiple shows different component of Valuation. Selecting any particular multiple might be biased at times in real world finance. The key drawback is that a firm can be overvalued and undervalued at the same time using different relative multiples. This reveals the key drawback of the process! How something is concluded to be Overvalued or Undervalued traditionally using any particular multiple e.g. Price to Book Value etc. in Valued based Investment is both conceptually and statistically flawed ! Infact , Value depends on the optimal combination of different multiples with right weightage and randomness component. Not just Static perspective of Cheapness means Value(which could be flawed in real world ) that rather higher order dynamics of how those Value ratios, their trajectories etc. The different ratios have their own dynamics over the time Higher Order Analysis not just First Order Analysis also affect the Value. There needs to be developed Proprietary models (AI/ML can also help) to Calculate True Value. That’s why I categorically repeat that the way Value has been traditionally and academically defined as Low Price to Book Value etc. is both foundationally as well as statistically flawed in Real World. Value is also relative on observer to observer. It’s NOT absolute ! \mathbit{V}\ =\ \mathbf{\epsilon}+\ \mathbit{k}\ \ast(\mathbit{EarningMultiple}\ \ast\mathbit{ProfiabilityMultiple}\ast\mathbit{GrowthMultiple}\ast\mathbit{MomentumMultiple}\ast\mathbit{SentimentMultiple}\ast\mathbit{MacroEconomicMultiple}\ast\mathbit{RelativeUtilityMultiple}\ \mathbit{of}\ \mathbit{the}\ \mathbit{Investor})/(\mathbit{PBVMultiple}\ast\mathbit{RiskMultiple}\ast\mathbit{CostMultiple}) Note : This is just a basic mathematical simple framework to show a framework but actually this is proprietary function to decide the Value and would depend investor to investor relatively. There could be more advanced Proprietary version in practical approach. Further, it’s not just first order thing but also how these multiples behave over the time also determine the Value over the time. So, What Value depends upon the combined approach for an investor. Conventional Academic Value itself needs Redefinition in the Real World. Momentum, Quality etc. are the components of the True Consolidated Value not different entities from them in Real World. All these aspects/components determine the Value not just Cheapness on just low Price to Book Value multiple. It’s an Open -ended rather than a closed ended formula in every scenario. Value is scientifically determined by the Causal forces affecting human (investors’) Relative perceptions in real life and Cheapness of Price is just one of the many causal factors. Contrary to conventional perceived notion that Cheapness means Value could be fundamentally misleading, Only Cheapness doesn’t mean Value to buy. There could be Cheapness but not Valuable overall. Moreover, Cheapness canine figured based on one Static Ratio. It’s has to be dynamically seen in Higher Order Perspective. These are proprietary aspects further. What is also alarming that Conventional flawed LS Regression type tools are applied to decide Cheapness not ! With slight shift of so called Line of Best fit due to error, something will be declared Cheap or not Cheap !! This itself is concern that how one calculates cheapness in real world based on such flawed statistical tools ! Key Result : Cheapness doesn’t mean Value always in Real World as widely believed !! Many factor dynamics below decide True Value in Real World Price to Book Value Multiple. Earning Multiples Debt Multiples Cash Flow Multiple Macro Economic Multiple Profitability & Growth Multiple Risk Multiple Cost Multiple Utility Multiple Momentum Multiple Sentimental Multiple Randomness Others factors This can be written as the statistical framework which remains proprietary here! Further, I repeat - it’s not just the first order rather how they have behaved dynamically over the time i. e. higher order would give better insight. So, Value in Real World depends upon many Causal Multiples which actually affects the Business of a firm and Human(Investor behavior) scientifically in a causal way as well as Randomness. It’s not just Book to value Multiple. Infact, this traditional academic definition of Value is incomplete and needs to be redefined. Also, by this traditional definition, if something is cheap doesn’t mean all the investors would buy it. Cheapness is just one of the factors to buy. The IMPORTANT Point is We need to Understand How Value should be defined and how it works actually in the Real World. One thing to Note that Momentum is also one of the Cause of Value . The reason being Momentum reflects the strength of Demand Supply forces being caused through Order Flow Imbalance. This force of demand supply also determines the value of something! Hence, contrary to traditional static absolute understanding of Value as different to Momentum, we have to understand it from the dynamic relative perspective that Momentum is itself one of the Causes of Value and Vice versa ! Infact, it’s two ways : Value causes Momentum and Momentum causes Value. It’s essentially like Laws of Nature/Physics where Value is like Total Energy( including Potential) and Momentum is like Kinetic form. So, all the other causal factors including momentum and randomness resultantly decide the Value. Infact, Value is the function of many different causal factors including Momentum as well as unknown randomness ! This has its origin in the Human behaviors driving the Investors’ behavior at large coming from the science and laws of Nature & Human Behavior. By the way, Scientifically using Laws of Stability Equilibrium in Nature , there also exists Proprietary approach to calculate the fair Valuation Relatively using Geometrical Methods! We shall possibly discuss this later as much we can(due to proprietary nature) after talking about how Markets are linked to the Laws of Nature and Physics ! Note : there are scientific approaches to calculate True Value based on force dynamics. This is Proprietary here now. 27.Geometrical & Stability Equilibrium Approach to Calculating the Fair Value : Apart from the above analysis, the Valuation can also be found out using Anlaysis as in Stability & Equilibrium analysis in Nature/ Science/Physics. That can be done using the Geometrical Analysis of the Trajectory curves, various Ratios etc.. and their degree of Stability & Equilibrium as well as done in Physics . Yes it’s like Laws of Motion in Physics but in a different form. The nature of motions are different but the underlying laws are similar…. But again this is Proprietary tool and can’t be disclosed here.. As I emphasize that by disclosing the exact content could influence the Investor & Traders Behavior and hence the Causality Parameters ! It could be the Part of Proprietary Strategies. The Stable Equilibrium Valuation can be estimated using Stability Analysis as in Physics based on the Trading Price data,Various Ratios data, trajectories etc. (assuming they reflect the public information) by applying the scientific analysis as in Physics e.g. Newton’s Laws of Motion. As explained earlier Things tend to revert to Stable Equilibrium Stage Valuation .It’s Laws of Nature. So, using the Laws of Nature, we can figure out that scientifically. The stocks at least Equilibrium stage are more likely to crash compared to high equilibrium stage stock. Higher Equilibrium Stage are supposed to be less risky . For example : It’s based on the Stability Laws in Nature & it can be applied to find out the Equilibrium stage /True Valuation scientifically in probabilistic sense… A simple explanation just for demonstration here as also explained by Prof Didier Sornette( ETH that Super Exponential Growth is likely to crash. The reason behind this possibility is explained using the Stability Equilibrium as I have proposed, the Super Exponential Growth of Stock suddenly would have likely less Stable Equilibrium degrees than Normal Natural Growth ! Now this is again Relative noting absolute ! What is unstable growth and what is Natural Growth would be figured out by the Scientific Stability Analysis and this is all relative ! This is Causal approach to the Valuation. Geometrical approach can be great Source of Practical Analysis even for Valuation ,Causal Dynamics etc. It’s proprietary here now. 28.How Machine Learning and others could help this Scientific Causality further : Although we know that there exists scientific laws which drive the markets backed by human behavioral forces but there definitely exists some deterministic patterns out there despite randomness. That hidden patterns could be deciphered further using Machine Learning to anayze Human Behavioral Dynamics ! I again categorically mention that Human Behaviors also follow the Scientific Laws of Nature like in Physics ( This is not the focus of this paper here.will write a detailed paper on that further) . But the summary is that to understand the Markets , We have to study Human Behaviors which consist of determinism and randomness both as in the Laws of Nature. That need to be further found out using ML etc. as that would essentially lead to the Proprietary Investment Strategies. Hence Can’t be completely disclosed here. There definitely exists some probabilistic patterns(yet Deterministic)in the measurement quantity of Force driven by Human Behaviors…That can be discovered using ML techniques. This is Proprietary discovery as of now can’t be publicized as it would affect the Causal Parameters! Infact the interesting part is that the amount of Content disclosed in an influential Research / News Paper could also affect the Human Behavior and Causality parameters in the Market !! Hence, I have to be careful now much I can disclose here as Causality forces is dynamic and driven by Human Behavioral Forces. Hence that’s left to the Reader to explore and do research on their own and come up with proprietary investment strategies. I have certain limitations to disclose it them beyond a limit in the paper. 29).Causal Equation of Market Let’s understand this in simple Newton’s Laws of motion like relationship in detail later but before that I write the Causal mechanism in the form of Ornstein Uhlenbeck(OU) mean reversion Process where Stock price tends to revert to their true mean value over the time which is the state of equilibrium. This is Pure Scientific. So in this equation, Net Effective Valuation at time t- V_{t_-}cause the momentum at time t, MOM_t to move. MOM_t≔Vt- Valuation at time t V_t is caused by the historical Momentum MO{M_t}_- as well. V_t\ ≔ MOMt- But unlike interest rate dynamics here the mean reversion level keeps on changing over the time for a stock or market depending upon various valuation constituents. dS_t=v_t\left(V_t-S_t\right)\thinspdt+\sigma\thinspdW_t v_t is the velocity at time t causing the momentum MOM_t=m_t\ast v_t V_t is the Stable Equilibrium State True Value Equivalent Calculated at time t. This will be calculated based as discussed above and results from the scientific dynamics of the forces. NOTE : What is extremely important to understand is that Even the Causal Equations have Error & Randomness Components which is the origin of Black Swan type events/ risks! The typical parameters and , together with the initial condition , completely characterize the dynamics, and can be quickly characterized as follows, assuming to be non-negative: b : "long term mean level". All future trajectories of will evolve around a mean level b in the long run; a : "speed of reversion". characterizes the velocity at which such trajectories will regroup around in time; σ : "instantaneous volatility", measures instant by instant the amplitude of randomness entering the system. Higher implies more randomness The following derived quantity is also of interest, σ^2 / 2a : "long term variance". All future trajectories of will regroup around the long term mean with such variance after a long time. 30.).Scientific Background of Convexity & Randomized Approach in finance : Quantum Energy Dynamics Connection Convexity : More Gain than Pain from a Random Event. The performance curves outward, hence looks "convex". Anywhere where such asymmetry prevails, we can call it convex, otherwise we are in a concave position. The implication is that you are harmed much less by an error (or a variation) than you can benefit from it, you would welcome uncertainty in the long run. The Picture taken from Internet : Source Nassim Taleb Papers Let’s look at how Convexity, Randomized Tools like RCT, Algo wheel etc. are Scientifically placed for Finance, Life etc.. As we know Feynman’s Path Integral approach has been the most important and powerful scientific result in Science which originated from Paul Dirac’s Principle of Least Action. So, as we know that Market is fundamentally a Quantum wave driven by Human Brains’ Behaviour, it would follow the Laws of Quantum world. Market would have uncertainty, randomness which could make it Unpredictable in some perspectives while they would follow some deterministic approach in other perspectives. These two aspects would co-exist simultaneously. One will have to discover that Determinism inside Randomness/Uncertainty at some scale. So, rather than directly trying to predict the Uncertainty, one has to look at Uncertainty from indirect perspectives. One has to find out the hidden Causal Mechanism inside Randomness. These all are being done while acknowledging the existence of Uncertainty at some scale. So, one may not directly predict Uncertainty at local scale or in individual perspective but one can definitely do that in global and collective context relatively. This is directly how Quantum world works. One might not predict the Behavior of a Quantum entity like electron individually due to Uncertainty but definitely as the collective system. That’s where the role of Randomized based approach fundamentally comes in. The most Predictive approach tries to predict at individual level but that’s not how the Quantum world works. They don’t acknowledge Uncertainty inherent in Nature. So, Scientifically one should find out global Certainty in Uncertainty/Randomness rather than trying to predict the Local Uncertainty! That’s what I had stated earlier that Randomness is NOT Randomly Random rather Deterministically Random .Randomenss also has cause. The discovery of that Determinism inside Randomness is the key to success in markets or real world or life. Hence, One should conduct Random Trials in Causal way and discover the Determinism inside. It’s like Path of Least Action as in Feynman’s Path Integral or Paul Dirac’s Principle of Least Action . That Path inside Randomness is the key. And random trials should be based on some Causal aspects not Randomly Random Trials always because for every Random Trial there would be some cost involved. One has to minimize that cost too. This can also be looked at from Reproducibility Crisis Perspective where one generates large number of computer random trials to select the best to produce the backtested results in a biased way !! The point is every random trial comes with some cost involved not free in real world ! So, by following the Causal adjusted Random Trials, own can maximize Convexity. This is also linked to the law of energy. If the cost is not minimized, due to herd behavior of energy in markets like interconnectedness in the complex system , it could worsen leading to huge disastrous Tail risk like Black Swan . Hence , the clue is to take the small loss before the energy dynamics leads to escalation and lead to huge tail risk. So, Convexity as the process of taking small loss by cutting the tail and keeping the upside open is fundamentally based on the Energy dynamics in the Quantum System driven by Humans. . As earlier explained that this complexity dynamics leads to instability and system crash. This is also related to Reproducibility Crisis in someway where one tries so many computer simulations and random trials to select the best ignoring that best is not always the best ,best could also be random !! . So, in my humble views, it’s not basically Knowledge Vs Convexity rather Knowledge+ Convexity . Through Causal knowledge one can increase the Convexity by minimizing the cost of random trials. And this is also linked to Reproducibility Crisis where just taking random trials and selecting the best blindly could be biased as done in Backtesting etc.. ! This is because even the Best could behave randomly over the time. That’s the clue to success : Discover the Determinism inside Randomness and follow the Path for Least Action . That’s based on Scientific Quantum Laws.. So, the concept of Convexity is scientifically linked to Laws of Science/Nature but at the same time by understanding the Causality Dynamics of Quantum Human Forces in the Market would help in the minimization of cost of Randomized Trials which would in turn improve the degree of Convexity !! So, Causality Dynamics should be seen as the Complementary to the Convexity rather than Competitor ! 31)Relativity in Antifragility /Convexity : The whole system can't be antifragile at the same time. It's like Zero Sum Game . For one entity to be Convex, other has to be Concave. So, In context of Anti-Fragility, One entity in the whole system like Markets can be Convex but it will be at the cost of some other entity being Concave ! Concept of Convexity: Antifragility as the Zero Sum Game in Markets But I would like to add here that : Despite the effort one must make to attain Convexity, system wise, it’s zero sum game. One entity say in Market would attain Convexity at the cost of other entity being Concave. It also points that Concept of Antifragility is Zero Sum Game at System level. One Part of the system can attain Antifragility, but other part will have to be Fragile especially in context of markets. Hence, one has to understand why everything can’t be made public as if every one tries to attain the Convexity, the concept of Convexity could fundamentally collapse. That means all the entities can’t be Convex at the same time in Markets to be able to run the market as a system. It’s like paradox : One should individually try to attain Convexity but at the same time all can’t attain the same. It’s like in Nature, where everything can’t be positive. Positive can exist only iff Negative would exist simultaneously . Convexity for someone is Concavity for another. Antifragility for one would be fragility for other. It’s Zero Sum Game & Relative. There can’t be absolute for all in the system e.g. Nature/ Markets, Life etc. . Hence, Convexity/Antifragility attained by one party would lead to Concavity/ fragility for another... That’s how Nature works. Positive and Negative co-exist Relatively to let the system function and everyone must try to be positive but it also Reality that everyone can’t be positive/good a social system/ markets/ Nature. 32.Complexity (Causality):Black Swan : Convexity : Scientific Causal Dynamics of Market Crash The real world we live in including markets, life ,Nature is basically a complex system . One should imagine essentially this as the energy system. In such system, inter connectedness among the human leading to herd behavior leads to fat- tailed type events like Black Swan. When there is sudden panic in the energy system through external or internal or both combined, there is formed a sudden cluster due to mutual energy attraction leading to instability in the system. This intrinsic instability leads to crash.It’s technically force and energy dynamics. This has to be causally understood in terms of energy dynamics. But that doesn’t means one can predict such with certainty but yes understanding them can help to detect the cluster energy formation as early as possible and in turn minimize tail risk. The key idea is that in Complex systems, one needs to understand the energy dynamics internally and hence to cut the Tail risk, early action would be quite useful. This is quite Causal dynamics like in Quantum Physics. But despite the Causal understanding, one can’t Completely make it Deterministic/Predictable. That’s where the role of Convexity comes into play. One has to remain convex to the Unpredictability . But by knowing the Causality and the scientific Internal energy dynamics of these Complex systems like Markets, Nature, Life, the cost of of attaining Convexity can be minimized. The cost of multiple trials can be minimized by selecting the Causal Adjusted Random Trials. Let me also explain the theory by Didier Sornette where there exists causality that super exponential growth can have downside burst risk. This is supported by the Laws of Energy & Nature. As explained earlier, like the Law of Gravity in Classical world, similar Law of Gravity exists in Human Behavior Dynamics as well in Quantum Economic space-time. Hence, as an example where some Physicist turned Economists have claimed about Super Exponential Growth leading to Crash. This has scientific base. But it must NOT be misunderstood. It doesn’t mean one can Deterministically predict the Bubble Crash always. In my view, The term Super Exponential Growth is not Absolute rather Relative here contextually !! It is scientifically based on Law of Energy but indeed Uncertainty exists as to when it would crash !! There can’t be Time Certainty over this in Real World always. This is also against the Law of Nature where there would be completely deterministic & Predictability of Crash ! 33). Causal -Adjusted Randomized Factor Approach Now having understood the scientific concepts now deduce the approach for Causal Factor Investing. We see that Market participants keep on finding new and new Deterministic factors and try to predict it using LS Regression type tools. They try to predict individual factors and try to find out the Ultimate complete set of Deterministic factors to predict the markets. That holy grail of such factors is utopian and scientifically could not possibly exist due to Quantum Nature of Market. The effort to make the Real World Deterministic which is fundamentally Quantum and having Uncertainty is against the Laws of Nature scientifically. Hence , that utopian dream of discovering the ultimate set of Deterministic Factors is against the Laws of Nature. It’s analogous to the ground breaking Godel Results in Mathematics and Logicwhere the effort to make market Deterministic by Complete set of Deterministic Factors is like making Arithmetic a Complete Axiomatic system while being Consistent! In other words, discovering the ultimate Holy Grail of Such Deterministic Factors is like trying to assume the world as Deterministic, which is scientifically not possible and against the Law of Quantum world. The key is to follow Forward Looking Causal Based Randomized Factor Approach unlike Backward Looking Conventional Approaches. So,In this approach like Feynman’s Path Integral & Paul Dirac Principle of Least Action, we try to apply the same in Human Quantum World. This is where we try to discover the hidden Determinism inside Random Causal Factors. This way we explore Certainty inside inherent Uncertainty. That’s the key and analogous to the Law of Quantum world (Physics being just a subset of that ). RCT, Algo wheel are based on that principles but there are many other tools using which we try to study the Determinism inside Random Factors. So, like an electron, we may not predict exactly about every single factor deterministically rather obviously in collective sense. It’s like Quantum world Interference, where one may not predict the trajectory of each electron individually , but definitely as a whole we can find some determinism that they would like around. Hence, unlike traditional approach of selecting factors on one by one basis, the scientific approach should be to Randomize the Factors based on some Causal understanding and create a Convex System. For that, On can follow the new approach Causal Based Randomized Learning Algorithm in the forward looking approach in the combined way rather than conventional backward looking approaches as usually followed in the AI/ML/RL techniques. This is for general demonstration purpose only here. Proprietary version would be more specialized for real world applications. For this new AI/ML Learning algorithm needs to be trained Random Selection of the Set of stocks based on the Causal based Deterministic and Random Factors according to their Real-World Valuations as discussed above S_1,S_2,S_3\ldots\ldots.S_n. Allocation of weights to them have to be dynamically set in a well diversified manner, putting various constraints on the weight, while also looking at Tail Risk perspective idiosyncratically. The key is to discover the Causal determinism in forward direction of time out of Randomized approach over the time follow the path of least action as a group as the scientific principle works. Causal adjusted Random Paths. Each Random Path will be allocated weightage allocation in the portfolio based on the degree of Causality. Like in Feynman’s Path Integral approach action over the path is taken into consideration to allocate probability. So, Allocate causal adjusted random weights to them at start to these selected Causal stocks w_1,w_2,w_3\ldots\ldots w_n. Some Predictive views based on Causality Factor Analysis could add up to weightages. Say for example if the S_1 has better risk-return profile expectation Causally in future than S_2, then even though randomly assigned weight w_1>w_2. Put Constraints on the Weights to make it well diversified for example w_1+w_2+w_3\le a\ risk management point of view. Based on the discovered determinism inside their random trajectories over the time, keep on adjusting weights dynamically to each of them over the time\ \ \ w_{t+1}=w_t+f(\alpha)\ast r_t.(for example just for demonstrative purpose here) There will be more sophisticated equations to update the weightage for practical applications. Randomized Trials would be done based on Causal perspectives as well in forward direction of time. The weights keep on adjusting dynamically based on discovering causal determinism inside their randomness in real world . Proprietary here. g) Effectively, the Portfolio becomes relatively Convex to handle Black Swan Risk Effectively and revert to their equilibrium true state over the time…. The exact equations and functions to update weight would be changing in real world scenarios depending on the scenarios and one’s own risk-return objectives. Here it is just for demonstration perspective. The key aspect of Causal Adjusted Randomized Factor Approach is to Randomize the Factors selection in causal way and then discover the Determinism hidden inside over the time. This is unlike traditional approach where selected traditional associational factors are bet in deterministic way. But to acknowledge that the Holy grail of Completely Deterministic Factors won’t ever be found due to Quantum behavior of Markets. The key is to discover Causal Determinism inside Uncertainty/Randomness ! This above new Randomized Approach is basically Causal Adjusted Randomized Trials like in Feynman’ s Path Integral and Paul Dirac’s Principle of Least Action in Science. This is Quantum Human Version of that here. RCT/Algo wheel are some basic tools based on this approaches. The more advanced proprietary scientific tools based on this approach could be applied. It’s also based on the Convexity Principle to build the Portfolio dynamically over the time. So, here True Risk is Black Swan Tail Risk, Normal Volatility should not be blindly taken as Risk rather they could be the essential scientific natural mechanism of the Markets ! As long as one is Convex and Prudent to Black Swan/Tail Risks, Intermediary Volatility should not be of much concern as according to the Laws of Nature and hence Markets, it would revert to the true Equilibrium Value state over the time. Beyond that it would it would influence the causal behavior hence the that would remain proprietary and private As explained in the beginning of the paper that there is Self-referential issue in the paper where publicly disclosing everything could influence the causal mechanism in someway …Even the information in this paper could be discounted by traders/investors in markets directly or indirectly. 34) Understanding Risk Mechanism in Real World: Fundamental Issues with Conventional Risk Management Tools e.g. Max Drawdown, Sharpe Ratio etc….. How Risk is often miscalculated ! It's often found that managers use conventional tools like Max Drawdown, Sharpe Ratio, using Backtesting etc. but let’s ask do they really measure the true risk in Real World ? In the fat-tailed financial world , true aspects of risk are in the tail side which come unpredictably. The fundamental problem with all these traditional tools is that they are often calculated on historical data in back ward direction of time based on observed data set. As explained above in various other contexts, direction of time is crucial in real world applications. These tools like Max Drawdown,Sharpe Ratio all are measured based on the data which have actually occurred in past. Say for example a stock had max drawdown of 25% in past. How this drawdown is calculated ? At time T = t, looking at the previous actual data (one that has actually happened!) one calculates the Max Drawdown. Or someone backtests a strategy at Time T= t based on the actual trading data for T