It seems like this is an example of Dual Beta:
In investing, dual-beta is a concept that states that a regular, market beta can be divided into downside beta and upside beta.
Essentially the beta (and intercepts) are calculated separately for up periods and down periods. So in your case, you seem to have an upside beta of slightly more than one but a large downside beta.
Note that a more modern technique is to calculate a portfolio's risk with respect to multiple factors rather than just a single overarching market factor. Those factors might give you more insight into exactly why your downside beta is higher. For example, you might have a high "volatility" factor, and downside changes tend to be larger (hence larger volatility). Or you might have a higher sensitivity to market size, where smaller-cap stocks might see larger drops in down markets. But calculating those factors is not trivial and is generally limited to large institutional investors.